Re: [LAU] More recent kernel config options

On Sun, Jul 26, 2009 at 02:24:25PM -0700, William Weston wrote:
> On Sat, 25 Jul 2009, Brent Busby wrote:
>
> > In addition to the question I posted earlier about PREEMPT_RCU, I've
> > found still other kernel config options that are not covered in most of
> > the extant howtos for setting up low latency kernels, because they are
> > recently added in the unpatched kernel. (Or at least, they're more
> > recent than most of the howtos...)
> >
> > I've been researching these options, mostly by googling (and
> > googling...and googling) the Linux Kernel Mailing List archives, and
> > also by looking at the config used by 64Studio, since it seems the
> > prevalent opinion is that they are stable as a rock. With the latter
> > approach of checking against 64Studio's config, I've had little luck
> > though, because their distro currently comes in a 2.1 "stable" version
> > that is based on Debian "etch", with a 2.6.21 kernel that predates the
> > existence of some of the config options I'm asking about; and a 3.0
> > "beta" version that uses a newer kernel, but should also probably be
> > taken with a grain of salt, because 64Studio is still working the bugs
> > out of it, and because it seems to have options turned on that the LKML
> > archives have warned strongly against. So, most of the information I've
> > gleaned has been from the LKML.
>
> I tried out the 64Studio stable version last fall, and had a lot of troubles
> with it. The 2.6.21 based kernel wouldn't even boot on my box, so I ended up
> booting up with the 2.6.24 based -rt kernel that I was using on Fedora 8 /
> CCRMA. There were too many libraries and apps that were way too out of date
> for my tastes, so I abandoned the whole experiment.
>
> > I might as well collate these questions here, for what good it may do.
> > The kernel I'm configuring is 2.6.29.6 with matching RT patches (-rt23),
> > though all of the options listed here are actually present even in the
> > unpatched 2.6.29.6 kernel.
> >
> >
> >
> > PREEMPT_RCU :
> >
> > This was the one my original question was about in the earlier post.
> > In the newer kernels (even mainline ones, without any RT patches), there
> > is a choice of "RCU Subsystem", with one option being "classic", and
> > other being "preemptable". The choice would seem obvious for low
> > latency, except that the help text warns that a preemptible RCU is
> > likely to expose serious kernel bugs that may render the system
> > completely unstable. (This is *not* the same setting as the various
> > CONFIG_PREEMPT options that have been present in the mainline kernel for
> > awhile now. This is something new, and it appears in menuconfig as a
> > separate setting.) I think I've mostly gathered from reading through
> > the whole process of debugging that the kernel gurus went through that
> > as of last year or so, this is now considered mostly stable, after
> > having been subjected to a utility called 'rcutorture', and having fixed
> > many lockups. I'd still be interested in anything anyone else can
> > contribute about it though, especially if they're using it and they
> > also think it is stable.
>
> There was a time when PREEMPT_RCU was unstable, and most of the -rt kernel
> releases would fix one RCU bug while adding another. These times are now past,
> however. PREEMPT_RCU has been completely stable for me on AMD and Intel (both
> multicore) for about a year now.
>
> > GROUP_SCHED :
> >
> > This one is interesting, and I don't know what to make of it, other than
> > that the LKML seems to have decided in the last two months or so that it
> > slows your system down and makes latencies worse.
> > The thing that's confusing about it though is that it's
> > described as a mechanism for grouping high priority tasks by group.
> > It's implied (though not spelled out specifically) that they even mean
> > by this Posix groups, because in one document I read, it says that
> > enabling this will cause you to be unable to get realtime as a non-root
> > user unless you are setup in a group specified in limits.conf. Hmm!
> > That sounds an awful lot like what we've just been calling pam_limits
> > for years now. Are we doing this with a kernel config option now? (One
> > which apparently doesn't work?)
> > The 3.0 (beta) version of 64Studio turns this on. Then again,
> > looking at their kernel config, they seem to turn everything on, and
> > that might be why it's considered beta. (The 2.1 stable version of
> > 64Studio seems to have a kernel old enough that it never had that
> > option as a choice.) Anyone have any feedback about this? It's only in
> > the past two months that the kernel gurus have decided that it's bad,
> > but they're actually considering marking it broken from what I've read.
>
> Here's another vote for completely broken! Every time I set up a box with the
> -rt kernel, I start with the distro's .config and tweak from there. And every
> time, I can't make realtime low-latency audio a reality until I disable
> GROUP_SCHED.
>
> > CGROUPS :
> >
> > This sounds cool, but I'm reasonably sure it's not actually necessary.
> > Documentation I've read suggests that this allows for letting
> > applications (or users) define CPU and memory pool affinity for tasks,
> > so that one could arbitrarily tie down particular threads or tasks to a
> > given processor core, or region of memory (or something like that).
> > However, the thing that makes me somewhat sure I don't positively need
> > this is that the same documentation also says that if you want to use
> > it, you need to create a new subdirectory under /dev, mount a new
> > pseudofilesystem under it, and then this module will populate that space
> > with dynamic configuration data about these affinity groups for running
> > tasks. I have neither seen any distro (even ones made for musicians)
> > that has set any such thing up out of the box, nor have I ever seen a
> > realtime howto that tells people to do it themselves. It sounds like
> > there's a lot of infrastructure necessary for this that's not common in
> > distros yet. (Though I see that regular Debian "lenny" turns this on in
> > the kernel without actually providing the special /dev support for it,
> > which I presume they're thinking you'll setup yourself if you're
> > interested.) Still, comments welcome...
>
> As far as I can tell, CGROUPS doesn't affect realtime latencies enough to make
> any sort of difference for realtime audio. It does, however, mess with the
> scheduler, even without setting up the affinity groups. I've noted some
> strange scheduling behavior: When running multiple instances of the same CPU
> bound task (like burnP6, mprime, dd, or your favorite stress tester), the tasks
> usually start off fairly evenly distributed between the cores, and then the
> affinity seems to drift, lumping these tasks on one half of the cores while
> leaving the other half mostly idle. On a core2quad with CGROUPS enabled, I
> generally have to run 12 instances of burnP6 to keep utilization of all four
> cores pegged at 99-100%. I wouldn't go as far as to say that CGROUPS is
> broken, but I fail to see the usefulness on a production audio system with the
> IRQ and other realtime priorities tuned properly, especially now that the core
> scheduling features for -rt are stable.
>
> > --
> > + Brent A. Busby + "We've all heard that a million monkeys
> > + UNIX Systems Admin + banging on a million typewriters will
> > + University of Chicago + eventually reproduce the entire works of
> > + Physical Sciences Div. + Shakespeare. Now, thanks to the Internet,
> > + James Franck Institute + we know this is not true." -Robert Wilensky
>
> I've also run into a handful of config options that either help or hurt
> realtime performance, depending on the specific hardware:
>
> NO_HZ (seems to work properly on most hardware these days)

Is that advised? I was told to use HZ_1000 in order to get MIDI to work correctly. Should I be using NO_HZ instead? I've got several machines to configure: a 32-bit Atom EEE, a 64-bit Intel Micro-ITX, and a 64-bit Asus Intel Core2Duo

> PCI_MSI (some PCI devices seem to add more latency with MSI)
> RTC_DRV_CMOS (this depends on your particluar RTC hardware)
> HPET_EMULATE_RTC (this depends on your particluar HPET hardware)
> X86_ACPI_CPUFREQ (creates TSC drift between cores on Intel)
>
> And of course, there's all kinds of profiling, stats, testing, and debugging
> options that make any realtime load suffer. Occasionally, you'll find a device
> driver that's just plain bad for -rt performance, but I've been running into
> this problem a lot less these days.
>
> Equally as important for low-latency are the scheduling priorities. After a lot
> of google, lkml, and trial and error, I've settled on the following for
> rock-solid low-latency audio:
>
> 99 FF migration
> 99 FF posixcputmr
> 98 FF IRQ-8 (realtime clock)
> 97 FF audio IRQ (in some cases means ieee1394 or specific USB port)
> 80 RR JACK
>
> All audio and MIDI apps should run at a lower realtime priority than JACK. All
> other IRQ- and sirq- threads should be set to
> SCHED_OTHER. To set this up, I've added the following to my /etc/rc.local:
>
> ---
> # set all irq threads to sched_other
> for irq in `pgrep 'IRQ-'`; do
> chrt -o -p 0 $irq;
> done
>
> # set all softirq threads to sched_other
> for sirq in `pgrep 'sirq-'`; do
> chrt -o -p 0 $sirq;
> done
>
> # set high prio IRQs
> chrt -f -p 98 `pgrep IRQ-8` # rtc
> chrt -f -p 97 `pgrep IRQ-16` # hda-intel
> ---
>
> And of course, there's those pesky BIOS options. Some of the newer Intel CPU
> features like to generate interrupts that the kernel can do absolutely nothing
> about. I had to disable C1E, CPU TM function, execute disable bit, and one of
> the ACPI options (can't remember which one) to get my core2quad to work.
> Disabling video interrupts (only an issue on older hardware) always seems to
> help. AMD systems seem to require much less (if any) BIOS tweaking for -rt.
>
> Please, anyone, correct me if I'm wrong on any of this. Most of this knowledge
> is gained from four years of following -rt development and tuning -rt kernels
> on a variety of hardware. I'm always adding to my bag of tricks when it comes
> to tuning realtime machines ;-}
>
>
> Cheers,
> --ww
> _______________________________________________
> Linux-audio-user mailing list
> Linux-audio-user@email-addr-hidden
> http://lists.linuxaudio.org/mailman/listinfo/linux-audio-user
_______________________________________________
Linux-audio-user mailing list
Linux-audio-user@email-addr-hidden
http://lists.linuxaudio.org/mailman/listinfo/linux-audio-user
Received on Mon Jul 27 04:15:06 2009