Subject: Re: [linux-audio-dev] lowish-latency patch and toolchain
From: David Olofson (david_AT_gardena.net)
Date: Sun Jul 09 2000 - 04:14:57 EEST
On Sun, 09 Jul 2000, Khimenko Victor wrote:
[...]
> >> Sorry. It was said 1000 times already but I can repeat once more just for you:
> >> 101% reliability => HARD real time => RTLinux. Period. If you NEED 101%
> >> reliability then stop even THINKING about normal Linux already. It's NOT
> >> for you.
>
> DO> If Ingo's patch didn't provide hard real time, neither does RTL.
>
> Wrong. Ingo patch was cooked up with help of benchmark and DOES NOT have any
> strong design under it. Unfortunatelly THE ONLY thing that can be proved with
> benchmark is UNAVAILABILITY of hard real-time, not AVAILIABILITY of anything.
Design by itself doesn't guarantee anything either, as there is
always the chance of bugs and other problems. True, a small RT core
that preempts everything non-RT has a greater chance of working
correctly, but it's not foolproof by design.
As to benchmarking not being able to prove availability of hard RT:
point taken - *nothing* can be measured with absolute precision that
way.
Then again, as *true* hard real time cannot exist in the real world
due to the risk of hardware failures, the kind of probability data
the benchmarking provides may be as close as you can ever get anyway,
and it's certainly good enough for most real life applications.
> DO> It's just a matter of what kind of peak latency the application
> DO> requires.
>
> Not at all. It's matter of guarantee. RTLinux CAN guarantee real-time (modulo
> bugs) - it was designed for it. Ingo patch CAN NOT. It fixes places which is
> known (more or less known, raither less then more; but it's other sory) to be
> bad for latency. It does not guarantee yu anything - it's pretty possible that
> some other software/hardware configuration will trigger situation where (even
> with Ingo patches!) Linux will not respond for second or so. It's does not even
> gurantee that on some weird (for you but not for user) configuration we'll not
> get such drop-outs for every minute or so. It's just quick hack "to make it
> wotk here and now" - nothing more.
As to guarantees; what prevents a particular mainboard design and/or
configuration from giving RTL bigger latencies than the guaranteed
figures? RTL *is* affected by the hardware and configuration
(settings that affect the system bus usage), and the worst case
latencies are only approximate. However, the probability that the
timing will be very close to the guarantees is *very* high, and
deviation will be minor.
Assuming that the latency will never be higher than 50%-100% above
the peak latency measured on your hardware is probably safe, but even
the hardware is too complex for critical systems to operate closer to
the limits than that. The same figure for Linux/lowlatency are worse
due to the complexity of the code that may get in the way, but it's
not as if the two solutions play by totally different rules! The
numbers have more than one bit here. ;-)
> DO> ********************************************************************
> DO> The reliability is a function of the peak scheduling latency vs. the
> DO> maximum latency that the application can deal with.
> DO> ********************************************************************
>
> Yeah. And Ingo patch DOES NOT guarantee you peak sheduling latency. Neither 5ms
> nor even 1s. Want to see sluggish system with Ingo patch applied ? I can cook
> up kernel module to make such a system in minutes.
Of course. Install a bad RTL driver module, and your RTL application
will get serious timing problems as well. Everything can be broken in
one way or another.
Under RTL, you're not supposed to disable IRQs for long periods. In
the kernel, you're not supposed to spend more time than you need in
a system call, you're not supposed to hold locks longer than you
need, and so on. True, Linux allows a whole lot more code to screw
your RT timing up, but it's still not a totally different thing.
> Now take look on number of
> available drivers. Do you really assume there are NO drivers/filesystem/modules
> who will make system sluggish with Ingo's patch ? You are joking, right ?
> And if you can con induce user to not use ANY such "bad" module (or perhaps
> even program - I'm quite sure some stupid program can trigger high latency
> as well even with Ingo patch) then why the hell you can induce the usage user
> to use your own "patched for this great application" kernel ? Gosh...
It's a matter of how much you can guarantee, and at what cost. Sure,
users can ruin the RT performance of their system in various ways,
and it's probably a whole lot easier with lowlatency than with RTL,
but OTOH, who dares to run various closed source plugins in kernel
space, without even basic memory protection...?
This is not only about the theoretical performance of the kernel. A
kernel core dump is just another way to miss deadlines - and likely
to cause a lot more trouble than that!
> DO> At some point, the biggest problem will be hardware failures rather
> DO> than missed deadlines, so there can never be exactly 100%
> DO> reliability.
>
> Oh yeah. Just there are difference betweek more-then-5ms-latency-once-per-week
> and 1sec-latency-once-per-minute . Ingo patch claims to give you first thing
> while in fact in quite likely situation it'll give you second. RTLinux does
> not suffer from such problem.
Yes it does, just as anything running on a CPU (unless it's a cycle
accurate padded loop inside a dedicated DSP). The only difference is
that you don't get problems with it until you need to go lower than
same 10-100 µs latency, depending on the hardware. RTL is not magic,
just a small and very fast RT kernel that lets your threads preempt
anything, as long as the IRQs are enabled. Oh, and both RTL code,
bad Linux drivers and even user space code (older XFree86 servers for
example) can wreck RTL latency guarantees.
> DO> That's not really interesting to this discussion though,
>
> It IS interesting. The more discussion progresses the less likely standard
> linux looks like good platform for your application. First you need 5ms latency
> in application without disk I/O, then you need 5ms latency for network
> application, now some developer request 101% reliability. It looks more and
> more like request for RTOS to me. And Linux is NOT RTOS. For better or for
> worse it's not RTOS and not trying to be RTOS. Larry's initial decision of
> request ("Other operating systems offer "soft realtime" and we don't want to
> port our code from that to the RT Linux model" and translation: "other
> operating systems have made poor design decisions, let's try and pressure
> Linus into doing the same thing with Linux") looks more and more justified.
Well, the alternative is adding protected memory to RTL, porting lots
of drivers to RTL, constructing something that resembles security in
the new environment, and finally perhaps hack the glue code to get it
to load and run SCHED_FIFO applications somewhat transparently. (The
last step is optional, but inventing a new API and compling
applications as kernel modules seems like a bad idea.)
Now, what looks like the worst design decision? Unless the latency
problems can be solved in a less uggly way, the only sane option
seems to be managing lowlatency as a separate patch. Mine and other
audio hackers' experience is that the RTL way is way too complicated,
and the orders of magnitude performance improvement over lowlatency
is not needed, and therefore not worth the effort. This may change,
however...
> DO> so lets return to real life: Thorough testing of Ingo's
> DO> patch has been done,
>
> Yeah ? With how much systems and how much @##$$% drivers, third-party modules
> (like softmodem ones) and how much weird programs (like VMWare) ? I repeat:
> if you can control other parts of user environment then why you can not control
> his choice of kernel ?
Well, I'd tell him to use a lowlatency patched kernel in that case.
Running dozens of thirdparty audio plugins under RTL reminds too much
of Windows; even though the performance is limited only by the sound
cards, the emulation of M$ DirectX Stability(TM) is not a feature I
desire. Besides, RTL wouldn't save him from poor or misconfigured
hardware anyway.
> DO> and the results indicate that the timing is deterministic enough to be
> DO> considered hard real time, for all practical matters.
>
> Let me not believe in it. I'll put this 8 years old IDE without DMA support
> and weird timing requirements (so you can not even use unmask irq hack) in my
> 700MHz Athlon system and your 5ms goal is way out of reach. And if you can
> control THIS aspect of system then why you can not control user's kernel
> choice ?
This problem exists for the established multimedia platforms as well,
and guess what: They suffer h*ll of a lot worse from that kind of
stuff than does Linux/lowlatency!
Now, RTL will allow the RT thread to run nicely under these
circumstances, but here's a different issue: The RT thread blocking
the HD driver ISR would most likely interfere with the HD performance,
or even result in data loss, when the RT thread starts to eat some
CPU power.
> DO> The Linux/lowlatency peak latency is a lot higher than that of RTL,
>
> What is MUCH more important there are no guarantee.
Well, to be really picky, a real life guarantee doesn't mean that
nothing will go wrong; it just means that the probability of that is
very low. Usually so low that you can afford assuming that there will
never be a problem.
> DO> BUT the relation between multimedia and robotics timing requirements
> DO> is in the same order of magnitude! If you need RTL for "100%"
> DO> reliable RT audio a about 1000 schedules/s, what do you use to
> DO> control an industrial robot at 10000 schedules/s...?
>
> If it's true that it's the end of story - Linux will not suit you, go use
> RTLinux or some other OS.
The point is that RTL for RT audio is overkill by an order of
magnitude, which would logically mean that an OS that does a whole
lot worse than RTL can still be adequate.
2.2.x/lowlatency does the job in my experience, and as to the risk
of finding a system/config that causes trouble, I haven't seen one so
far. Granted, I've only used it on 3 different machines personally -
I might just have been lucky so far.
> >> Of course. It's easy. Use RTLinux. You need it anyway (as you said you need
> >> 101% reliability) so what's the point of discussion at all ?
>
> DO> Speaking for myself, the point is that
>
> DO> * Ingo's lowlatency patch *does* provide adequate quality real time,
>
> In lab (i.e.: in strictly controlled environment), not in real world.
Ok; just how much are production systems going to put up with?
It requires knowledge, experience, luck and black magic to get a
decent Windows DAW together, and the user must be instructed never to
install games and other non-audio applications; preferably never to
install anything at all on that box, or he *will* have trouble sooner
or later.
In the light of this, I'd say it's more than acceptable that the
average users of Linux RT multimedia should stay away from messing
with the kernel configuration, and not mess with low level driver
config files.
> DO> * it doesn't require porting code to a different environment,
>
> Yeah. "Other operating systems have made poor design decisions, let's try and
> pressure Linus into doing the same thing with Linux".
No. The reason I'm commenting on this thread is that there seems to
be some confusion regarding the difference between the kind of
applications that RTL and Linux/lowlatency are meant for, possibly
making people believe that it's totally impossible to find a solution.
> DO> * it doesn't break system security and stability entirely, as is
> DO> the case when the code has to run in unprotected kernel space.
>
> It just adds god knows how much petential deadlocks - not a big deal, really.
> "Lets add potentially dangerous code so millions will suffer and thusands will
> benefit". Does not look like good deal to me.
No, that's why a better solution would be nicer than maintaining the
lowlatency patch separately, just because it sucks too bad to go
mainstream. Do you think I want code that's too unreliable for
mainstream in systems that are supposed to deal with high end RT
audio? If it's likely to cause problems for "the millions", how could
the "thousands" stand clear of those problems?
> DO> Ingo's lowlatency patch *did* achieve the goal of taking the latency
> DO> to professionally usable levels, and even below the previously
> DO> "multimedia OS record" of 3 ms, held by BeOS.
>
> Benchmarketing, yeah ? Not @ lkml, please.
If it's any kind of marketing, it's of the idea of ms range RT in
user space - not of an implementation that's clearly not up to the
standards. If it can't be done right due to the design of the kernel,
we'll just look for another solution.
> DO> It even did so without any messy "direct access" API, or working around
> DO> the normal driver API.
>
> If it's so great for YOU then use it. If you push EVERYONE to use it... It's
> other story (see above).
Yes. Sorry for pushing a bad solution, if that's what I seem to be
doing.
[...]
> DO> Any guesses as to when it'll be definitely too late for this kind of
> DO> patches for 2.4?
>
> WHAT kind of patches ?
Quite a few people wish someone knew...
//David
.- M u C o S --------------------------------. .- David Olofson ------.
| A Free/Open Multimedia | | Audio Hacker |
| Plugin and Integration Standard | | Linux Advocate |
`------------> http://www.linuxdj.com/mucos -' | Open Source Advocate |
.- A u d i a l i t y ------------------------. | Singer |
| Rock Solid Low Latency Signal Processing | | Songwriter |
`---> http://www.angelfire.com/or/audiality -' `-> david_AT_linuxdj.com -'
This archive was generated by hypermail 2b28 : Sun Jul 09 2000 - 07:19:34 EEST