| Problem with large .txt file [message #55044] |
Tue, 31 July 2007 09:03  |
aleks.franca@gmail.co
Messages: 33
Registered: March 2007
|
Member |
|
|
Hy there!
I have a large txt file (6.384KB) with 65000 lines. It's very large.
I need to open it, read every line, replace the spaces for commas, and
write the result in an output.txt
The code is ok. The problem is that idl crashes on line 7208. There's
a memory problem there.
Is there any way to do for all the lines of the txt file?? I don't
know how to deal with memory problems. Sorry!
I'm doing like this
:-----------------CUT HERE--------------------
i=0
line=''
; Open the text file:
OPENR, inunit, '60mil_equal.txt', /GET_LUN
; Open the terminal as a file:
OPENW, outunit, 'output.txt', /GET_LUN
;================================
WHILE ~EOF(inunit) DO BEGIN
READF, inunit, line
line = STRJOIN(STRSPLIT(line, /EXTRACT), ', ')
PRINTF, outunit, line
i =i+1
ENDWHILE
FREE_LUN, inunit
FREE_LUN, outunit
END
:-----------------CUT HERE--------------------
Thank you very much
|
|
|
|
| Re: Problem with large .txt file [message #55089 is a reply to message #55044] |
Wed, 01 August 2007 16:20  |
badjelly.witch
Messages: 27
Registered: May 2006
|
Junior Member |
|
|
On Aug 1, 5:36 am, Conor <cmanc...@gmail.com> wrote:
> Of course, considering that your files have a .txt extension, it seems
> likely that you are on a windows machine, and therefore have no access
> to sed.
You're just trying to bait me, right?
|
|
|
|
| Re: Problem with large .txt file [message #55096 is a reply to message #55044] |
Wed, 01 August 2007 09:35 |
Jean H.
Messages: 472
Registered: July 2006
|
Senior Member |
|
|
Paul van Delst wrote:
> David Fanning wrote:
>> aleks.franca@gmail.com writes:
>>
>>> But at least I didn't have an IDL problem. That's what I was afraid of.
>>
>> My wife's big concern is that I'm going to try to explain
>> something to her.
>
> Huh. So it's not just me, then. Good to know.
>
> :o)
>
> cheers,
>
> paulv
Would it even be a constant for scientific programmers??????
Jean
|
|
|
|
| Re: Problem with large .txt file [message #55101 is a reply to message #55044] |
Wed, 01 August 2007 08:38 |
Haje Korth
Messages: 651
Registered: May 1997
|
Senior Member |
|
|
duh... H.
<aleks.franca@gmail.com> wrote in message
news:1185970907.269552.63730@q75g2000hsh.googlegroups.com...
> On 1 ago, 04:48, Wox <nom...@hotmail.com> wrote:
>> On Wed, 01 Aug 2007 02:09:10 -0000, "aleks.fra...@gmail.com"
>>
>> <aleks.fra...@gmail.com> wrote:
>>> The error message is that There's not enough space in disk.
>>
>> And you hard drive is not full?
>
> hy.
> Just now I realized that my routine is in my USB zipdrive, and it`s
> full. I thought I was working in the Hard Disk.
> I`m really sorry you all for the incovenience.
>
> Thanks
>
|
|
|
|
|
|
|
|
| Re: Problem with large .txt file [message #55107 is a reply to message #55044] |
Wed, 01 August 2007 06:04 |
aleks.franca@gmail.co
Messages: 33
Registered: March 2007
|
Member |
|
|
On 1 ago, 09:42, David Fanning <n...@dfanning.com> wrote:
> aleks.fra...@gmail.com writes:
>> Just now I realized that my routine is in my USB zipdrive, and it`s
>> full. I thought I was working in the Hard Disk.
>> I`m really sorry you all for the incovenience.
>
> You would get along with my wife. :-)
>
> Cheers,
>
> David
> --
> David Fanning, Ph.D.
> Fanning Software Consulting, Inc.
> Coyote's Guide to IDL Programming:http://www.dfanning.com/
> Sepore ma de ni thui. ("Perhaps thou speakest truth.")
But at least I didn't have an IDL problem. That's what I was afraid of.
|
|
|
|
| Re: Problem with large .txt file [message #55112 is a reply to message #55044] |
Wed, 01 August 2007 05:42 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
aleks.franca@gmail.com writes:
> Just now I realized that my routine is in my USB zipdrive, and it`s
> full. I thought I was working in the Hard Disk.
> I`m really sorry you all for the incovenience.
You would get along with my wife. :-)
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: Problem with large .txt file [message #55113 is a reply to message #55044] |
Wed, 01 August 2007 05:21 |
aleks.franca@gmail.co
Messages: 33
Registered: March 2007
|
Member |
|
|
On 1 ago, 04:48, Wox <nom...@hotmail.com> wrote:
> On Wed, 01 Aug 2007 02:09:10 -0000, "aleks.fra...@gmail.com"
>
> <aleks.fra...@gmail.com> wrote:
>> The error message is that There's not enough space in disk.
>
> And you hard drive is not full?
hy.
Just now I realized that my routine is in my USB zipdrive, and it`s
full. I thought I was working in the Hard Disk.
I`m really sorry you all for the incovenience.
Thanks
|
|
|
|
|
|
| Re: Problem with large .txt file [message #55126 is a reply to message #55044] |
Tue, 31 July 2007 19:17 |
aleks.franca@gmail.co
Messages: 33
Registered: March 2007
|
Member |
|
|
On 31 jul, 16:37, "Haje Korth" <haje.ko...@nospam.jhuapl.edu> wrote:
> True, the variable is not the problem. It will give a wrong line count but
> this is not cause for memeory error. Now I am puzzled. I am reading text
> files up to 100MB without problems. I am sure bigger ones work too. I
> fold... H.
>
> "Jeff N." <jnett...@utk.edu> wrote in message
>
> news:1185908328.461871.25230@m37g2000prh.googlegroups.com...
>
>> On Jul 31, 12:03 pm, "aleks.fra...@gmail.com" <aleks.fra...@gmail.com>
>> wrote:
>>> Hy there!
>>> I have a large txt file (6.384KB) with 65000 lines. It's very large.
>>> I need to open it, read every line, replace the spaces for commas, and
>>> write the result in an output.txt
>
>>> The code is ok. The problem is that idl crashes on line 7208. There's
>>> a memory problem there.
>
>>> Is there any way to do for all the lines of the txt file?? I don't
>>> know how to deal with memory problems. Sorry!
>
>>> I'm doing like this
>
>>> :-----------------CUT HERE--------------------
>>> i=0
>>> line=''
>>> ; Open the text file:
>>> OPENR, inunit, '60mil_equal.txt', /GET_LUN
>>> ; Open the terminal as a file:
>>> OPENW, outunit, 'output.txt', /GET_LUN
>>> ;================================
>>> WHILE ~EOF(inunit) DO BEGIN
>>> READF, inunit, line
>>> line = STRJOIN(STRSPLIT(line, /EXTRACT), ', ')
>>> PRINTF, outunit, line
>>> i =i+1
>>> ENDWHILE
>
>>> FREE_LUN, inunit
>>> FREE_LUN, outunit
>
>>> END
>>> :-----------------CUT HERE--------------------
>
>>> Thank you very much
>
>> I must be missing something, but I don't see you actually using the
>> variable i for anything. All you do is increment it each time around
>> the loop. Unless this isn't the final version of your code and you're
>> going to add functionality that uses that variable, you could take it
>> out.
>
>> Jeff
Hy Haje!
I can read the file without a problem. I've just tested it. But it
seems that IDL cannot handle Writing the file into another file. Here
is where the error shows up.
thank you
|
|
|
|
| Re: Problem with large .txt file [message #55127 is a reply to message #55044] |
Tue, 31 July 2007 19:09 |
aleks.franca@gmail.co
Messages: 33
Registered: March 2007
|
Member |
|
|
On 31 jul, 17:16, Bruce Bowler <bbow...@bigelow.org> wrote:
> On Tue, 31 Jul 2007 09:03:54 -0700, aleks.fra...@gmail.com wrote:
>> Hy there!
>> I have a large txt file (6.384KB) with 65000 lines. It's very large. I
>> need to open it, read every line, replace the spaces for commas, and
>> write the result in an output.txt
>
>> The code is ok. The problem is that idl crashes on line 7208. There's a
>> memory problem there.
>
> Crashes "how"? What's the error message? Is there anything unusual
> about line 7208, like embedded control characters or something else? (or
> maybe line 7207 or 7209?)
>
> Bruce
The error message is that There's not enough space in disk.
The lines 7207, 7208, and 7209 have nothing unusual. They are like
this
0.0903 0.0762 0.0744 0.1778 0.2024 13.2929 0.2531 0.4266 0.3123
3.6259 -1.4637 -6.1165 0.4098 2.3885 6 2.2416 79
0.0843 0.0701 0.0641 0.1842 0.1728 12.0156 0.1874 0.5139 0.2918
3.2822 -1.3031 -5.5164 0.4836 2.873 6 2.0511 100
0.0948 0.0792 0.0976 0.1681 0.2214 14.5702 0.3253 0.2956 0.3703
3.9688 -1.632 -6.7057 0.2653 1.7221 1 2.3352 13
The cells are separate by tabs.
I also used the function memory(), but I can't underestand it.
I have 1 GB of RAM. It shows like this:
IDL> print, result
646311 15200 14871 646311
It's weird, I can't see any problems with it.
|
|
|
|
| Re: Problem with large .txt file [message #55130 is a reply to message #55044] |
Tue, 31 July 2007 13:16 |
Bruce Bowler
Messages: 128
Registered: September 1998
|
Senior Member |
|
|
On Tue, 31 Jul 2007 09:03:54 -0700, aleks.franca@gmail.com wrote:
> Hy there!
> I have a large txt file (6.384KB) with 65000 lines. It's very large. I
> need to open it, read every line, replace the spaces for commas, and
> write the result in an output.txt
>
> The code is ok. The problem is that idl crashes on line 7208. There's a
> memory problem there.
>
Crashes "how"? What's the error message? Is there anything unusual
about line 7208, like embedded control characters or something else? (or
maybe line 7207 or 7209?)
Bruce
|
|
|
|
| Re: Problem with large .txt file [message #55131 is a reply to message #55044] |
Tue, 31 July 2007 12:37 |
Haje Korth
Messages: 651
Registered: May 1997
|
Senior Member |
|
|
True, the variable is not the problem. It will give a wrong line count but
this is not cause for memeory error. Now I am puzzled. I am reading text
files up to 100MB without problems. I am sure bigger ones work too. I
fold... H.
"Jeff N." <jnettle1@utk.edu> wrote in message
news:1185908328.461871.25230@m37g2000prh.googlegroups.com...
> On Jul 31, 12:03 pm, "aleks.fra...@gmail.com" <aleks.fra...@gmail.com>
> wrote:
>> Hy there!
>> I have a large txt file (6.384KB) with 65000 lines. It's very large.
>> I need to open it, read every line, replace the spaces for commas, and
>> write the result in an output.txt
>>
>> The code is ok. The problem is that idl crashes on line 7208. There's
>> a memory problem there.
>>
>> Is there any way to do for all the lines of the txt file?? I don't
>> know how to deal with memory problems. Sorry!
>>
>> I'm doing like this
>>
>> :-----------------CUT HERE--------------------
>> i=0
>> line=''
>> ; Open the text file:
>> OPENR, inunit, '60mil_equal.txt', /GET_LUN
>> ; Open the terminal as a file:
>> OPENW, outunit, 'output.txt', /GET_LUN
>> ;================================
>> WHILE ~EOF(inunit) DO BEGIN
>> READF, inunit, line
>> line = STRJOIN(STRSPLIT(line, /EXTRACT), ', ')
>> PRINTF, outunit, line
>> i =i+1
>> ENDWHILE
>>
>> FREE_LUN, inunit
>> FREE_LUN, outunit
>>
>> END
>> :-----------------CUT HERE--------------------
>>
>> Thank you very much
>
> I must be missing something, but I don't see you actually using the
> variable i for anything. All you do is increment it each time around
> the loop. Unless this isn't the final version of your code and you're
> going to add functionality that uses that variable, you could take it
> out.
>
> Jeff
>
|
|
|
|
| Re: Problem with large .txt file [message #55133 is a reply to message #55044] |
Tue, 31 July 2007 12:14 |
aleks.franca@gmail.co
Messages: 33
Registered: March 2007
|
Member |
|
|
On 31 jul, 15:58, "Jeff N." <jnett...@utk.edu> wrote:
> On Jul 31, 12:03 pm, "aleks.fra...@gmail.com" <aleks.fra...@gmail.com>
> wrote:
>
>
>
>> Hy there!
>> I have a large txt file (6.384KB) with 65000 lines. It's very large.
>> I need to open it, read every line, replace the spaces for commas, and
>> write the result in an output.txt
>
>> The code is ok. The problem is that idl crashes on line 7208. There's
>> a memory problem there.
>
>> Is there any way to do for all the lines of the txt file?? I don't
>> know how to deal with memory problems. Sorry!
>
>> I'm doing like this
>
>> :-----------------CUT HERE--------------------
>> i=0
>> line=''
>> ; Open the text file:
>> OPENR, inunit, '60mil_equal.txt', /GET_LUN
>> ; Open the terminal as a file:
>> OPENW, outunit, 'output.txt', /GET_LUN
>> ;================================
>> WHILE ~EOF(inunit) DO BEGIN
>> READF, inunit, line
>> line = STRJOIN(STRSPLIT(line, /EXTRACT), ', ')
>> PRINTF, outunit, line
>> i =i+1
>> ENDWHILE
>
>> FREE_LUN, inunit
>> FREE_LUN, outunit
>
>> END
>> :-----------------CUT HERE--------------------
>
>> Thank you very much
>
> I must be missing something, but I don't see you actually using the
> variable i for anything. All you do is increment it each time around
> the loop. Unless this isn't the final version of your code and you're
> going to add functionality that uses that variable, you could take it
> out.
>
> Jeff
the variable i is counting for other functionality.
I`ve heard about IDL working with blocks. Every time it processes a
block, it frees the memory and brings another block to process. Does
IDL do that with reading txt large files?
thank you for your helps!
|
|
|
|
| Re: Problem with large .txt file [message #55134 is a reply to message #55044] |
Tue, 31 July 2007 11:58 |
Jeff N.
Messages: 120
Registered: April 2005
|
Senior Member |
|
|
On Jul 31, 12:03 pm, "aleks.fra...@gmail.com" <aleks.fra...@gmail.com>
wrote:
> Hy there!
> I have a large txt file (6.384KB) with 65000 lines. It's very large.
> I need to open it, read every line, replace the spaces for commas, and
> write the result in an output.txt
>
> The code is ok. The problem is that idl crashes on line 7208. There's
> a memory problem there.
>
> Is there any way to do for all the lines of the txt file?? I don't
> know how to deal with memory problems. Sorry!
>
> I'm doing like this
>
> :-----------------CUT HERE--------------------
> i=0
> line=''
> ; Open the text file:
> OPENR, inunit, '60mil_equal.txt', /GET_LUN
> ; Open the terminal as a file:
> OPENW, outunit, 'output.txt', /GET_LUN
> ;================================
> WHILE ~EOF(inunit) DO BEGIN
> READF, inunit, line
> line = STRJOIN(STRSPLIT(line, /EXTRACT), ', ')
> PRINTF, outunit, line
> i =i+1
> ENDWHILE
>
> FREE_LUN, inunit
> FREE_LUN, outunit
>
> END
> :-----------------CUT HERE--------------------
>
> Thank you very much
I must be missing something, but I don't see you actually using the
variable i for anything. All you do is increment it each time around
the loop. Unless this isn't the final version of your code and you're
going to add functionality that uses that variable, you could take it
out.
Jeff
|
|
|
|
| Re: Problem with large .txt file [message #55136 is a reply to message #55044] |
Tue, 31 July 2007 11:06 |
Conor
Messages: 138
Registered: February 2007
|
Senior Member |
|
|
On Jul 31, 1:36 pm, Conor <cmanc...@gmail.com> wrote:
> On Jul 31, 1:15 pm, hradilv <hrad...@yahoo.com> wrote:
>
>
>
>> On Jul 31, 11:27 am, "Haje Korth" <haje.ko...@nospam.jhuapl.edu>
>> wrote:
>
>>> set i ti i=0l (make it a long)
>
>>> H.
>
>>> <aleks.fra...@gmail.com> wrote in message
>
>>> news:1185897834.196562.301440@57g2000hsv.googlegroups.com...
>
>>>> Hy there!
>>>> I have a large txt file (6.384KB) with 65000 lines. It's very large.
>>>> I need to open it, read every line, replace the spaces for commas, and
>>>> write the result in an output.txt
>
>>>> The code is ok. The problem is that idl crashes on line 7208. There's
>>>> a memory problem there.
>
>>>> Is there any way to do for all the lines of the txt file?? I don't
>>>> know how to deal with memory problems. Sorry!
>
>>>> I'm doing like this
>
>>>> :-----------------CUT HERE--------------------
>>>> i=0
>>>> line=''
>>>> ; Open the text file:
>>>> OPENR, inunit, '60mil_equal.txt', /GET_LUN
>>>> ; Open the terminal as a file:
>>>> OPENW, outunit, 'output.txt', /GET_LUN
>>>> ;================================
>>>> WHILE ~EOF(inunit) DO BEGIN
>>>> READF, inunit, line
>>>> line = STRJOIN(STRSPLIT(line, /EXTRACT), ', ')
>>>> PRINTF, outunit, line
>>>> i =i+1
>>>> ENDWHILE
>
>>>> FREE_LUN, inunit
>>>> FREE_LUN, outunit
>
>>>> END
>>>> :-----------------CUT HERE--------------------
>
>>>> Thank you very much
>
>> Or just use sed... Right tool for the right job.
>
> I was going to say use perl, but I suppose that's a bit simpler.
>
> sed 's/ /,/g' < 60mil_equal.txt > output.txt
>
> viola! Simple and elegant.
>
> Of course, considering that your files have a .txt extension, it seems
> likely that you are on a windows machine, and therefore have no access
> to sed. In that case, I would recommend learning perl, which can
> easily be installed on windows. A better option would be to just use
> linux or mac :)
Also, you could do this same thing in just about any text editor.
Even Microsoft Word has a find/replace option which would easily do
this. Notepad might too. I mention this simply because IDL really
isn't the best language for string parsing/editing. Other than that,
I'm afraid I'm out of suggestions. I don't see any problems with your
code.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
