Subject: Re: [linux-audio-dev] file sizes and retrofits
From: Paul Barton-Davis (pbd_AT_Op.Net)
Date: su helmi 06 2000 - 22:41:41 EST
>renders it to the graphic display. I assume it uses memory instead of
>disk, restricting the possible file size to RAM. Am I correct in
>assuming that this method is indeed restricting the acceptable size ?
well, if really expected to be able to copy the file into memory, then
yes, there is a limit: around 4GB. However, standard Linux ext2
filesystems only have files up to 2GB. Despite this, long before you
got close to 2GB, performance would be appalling. In addition, many
Unix environments, including Linux, sometimes impose limits on user's
resource use that would restrict this as well - check the limit(1)
command.
programs that do this directly, however, are plain naive. if you found
something like this:
/* get size of file */
stat (filename, &statbuf);
/* allocate memory to hold it */
buf = malloc (statbuf.st_size);
/* open the file */
fd = open (filename, O_RDONLY);
/* read the file into memory */
read (fd, buf, statbuf.st_size);
then you know you're dealing with this kind of naivete. Slightly less
naive is a program that uses mmap(2) to map the whole file into
memory:
/* get size of file */
stat (filename, &statbuf);
/* open the file */
fd = open (filename, O_RDONLY);
/* map the file into memory */
buf = mmap (0, statbuf.st_size, MAP_SHARED, PROT_READ, fd, 0);
However, as Juhana has pointed out to me, even this is a pretty weak
approach - Linux (and most Unices) have a 4GB limit on the address
space of a process (because they use 32bit pointers), so this won't
handle files larger than 4GB (and realistically, won't even handle
that, since some of the 4GB is already occupied by the program itself,
other data, shared libraries, etc).
If you want to work with large data sets in memory, which is a good
idea in general, you need to add at least one layer of abstraction to
things, so that the size limit goes away. Typically, you get a
"window" of some size through which you can "look at" the data, and a
way to move it; when you move it, the "layer" below that in the
software makes sure that the requested region is in memory before it
returns. Sometimes, that doesn't involve any work; other times it
could involve a huge amount of effort by the machine.
Many, many programs don't do this, however, just like most soundfile
editors don't use edit/play lists.
--p
This archive was generated by hypermail 2b28 : pe maalis 10 2000 - 07:23:27 EST