arrow left BLOG

Raspberry Pi real-time kernel [SD image available]

RT-TestsLinux kernel has configuration options that affect it’s real-time capabilities. Default Raspbian kernel is compiled with CONFIG_PREEMPT option that allows all kernel code outside of spinlock-protected regions and interrupt handlers to be preempted by higher priority kernel threads. Real-time patch from Ingo Molnar adds CONFIG_PREEMPT_RT option that allows nearly all of the kernel code to be preempted, except for a few raw spinlock critical regions.

 

The histogram shows latency comparison between the default Raspbian kernel and the kernel with real-time patch.

 

We’ve used cyclictest program to obtain data with the following parameters:

cyclictest -l10000000 -m -n -a0 -t1 -p99 -i400 -h400 -q 

 

Latencies for default Raspbian kernel (PREEMPT) in uS: Min: 00014 Avg: 00028 Max: 01247

Latency results for PREEMPT_RT patched kernel in uS: Min: 00012  Avg: 00027 Max: 00077

 

The most important characteristic here is the max latency which indicates the longest time it might take your Raspberry Pi to respond to an event. As it can be observed PREEMPT_RT kernel provides much lower latency thus making it more suitable for usage in time-sensitive embedded systems. It is important to remember that to make full use of real time capabilities of the kernel it is necessary to do proper prioritizing.

 

You can compile PREEMPT_RT real time kernel for Raspberry Pi by yourself or you can download modified Raspbian SD card image that we prepared. The image was tested for compatibility with Raspberry Pi Model B and Raspberry Pi Model B+. The image is not intended for general purpose use but rather for embedded electronics projects.

 

UPDATE (10 SEP 2014) – Raspbian with RT-kernel for Raspberry Pi Model B+ (also compatible with Model B)

UPDATE (25 NOV 2014) – Raspbian with RT-kernel for Raspberry Pi Model A/B/A+/B+. Download links:

UPDATE (21 JAN 2015) – Raspbian with RT-kernel for Raspberry Pi Model A/B/A+/B+. SD card image downloads:

This document is not updated, please proceed to docs.blog.emlid.com for latest version!

 

56 thoughts on “Raspberry Pi real-time kernel [SD image available]

  1. bennierex says:

    Great work! I’ll give it a go on my RPi later this week.

  2. Sys-Ak says:

    Wanted to try the image on my RPi, but i’ve stucked after boot. USB keyboard didn’t work, I couldn’t log in.

    1. It is probably because of the dwc_otg.speed=1 option int cmdline.txt that was added for better stability.
      This options seems to break compatibility with some of the keyboards. To fix this you can either change it to dwc_otg.speed=0 which is not recommended or you can find a compatible keyboard.
      You can find more information about this issue here – http://raspberrypi.stackexchange.com/questions/4090/how-can-dwc-otg-speed-1-be-made-to-work

      I’d like to note that this image is not intended for general-purpose usage, but for controlling embedded systems. I recommend using ssh to work with RPi, because you probably won’t have a keyboard connected to your plane or copter:)

  3. solenerotech says:

    Hi, could you please add some tips in order to add this modification by compiling it? I would not to start with a new sd.
    Thanks
    Oscar

    1. Oscar, there’s a second download link with a standalone kernel that you can try with your Raspbian, instructions are in the README file in the archive. If you want to compile the kernel yourself you can find a few tutorials over the internet, for example this one – http://elinux.org/RPi_Kernel_Compilation

  4. Please, can you explain me how I can use the files in the archive? Thanks, Silvio

    1. Silvio, if you mean the archive with SD card image then you need to write (not just copy) it to your SD card. You can find instructions how to do this here – http://elinux.org/RPi_Easy_SD_Card_Setup
      If you are talking about the archive with the standalone kernel, then you can find instruction in the README file inside the arhive.

      1. Yes I mean the image of the SD card, there are two other files into the archive. I know how to write the file with img extention, for the other? How I can use them?

        1. Silvio, the archive by the link “SD-card image of Raspbian with PREEMPT_RT kernel” contains only img file and readme. You don’t need the other archive.

  5. Cesar Sanchez says:

    Hi. I’m already testing with the full image, but i have a question.

    Can I make an update or it will uninstall the patch? Thanks, Cesar

    1. Hi Cesar,

      You can make an update, but after it make sure that kernel option in /boot/config.txt points to the rt-kernel:
      kernel=kernel-rt.img

  6. Aaron says:

    Hi, can you say how to get this working with the B+ model? I can get it to boot by updating the bootcode.bin and start.elf on the boot partition to the latest versions. But USB and network are not working after boot. Is an updated kernel required to support the B+ changes to these devices? Thanks for your work on this.

    1. Hi Aaron,

      It’s probably because Raspberry Pi Model B+ is using the newer version of the USB chip and it seems that kernel and modules have to be rebuilt using updated sources. We’ll make an updated image, but I am not sure how soon.

      1. Jonathan says:

        Mikhail,

        If I was to go ahead and take a stab at rebuilding the kernel with updated sources, would I just need to use the 3.12 rpi kernel instead of 3.10, and repatch it with the appropriate preempt_rt? Or would I need to pull in the drivers for this specific USB hub?

        I’ve started to do some tinkering, but the realtime patch wasn’t playing nicely and so I didn’t get very far. Let me know if you have any tips for getting this rolling – I’d be happy to share an image once I get it done.

        1. Jonathan, no, I don’t think that you need to manually add the driver, I’m sure that the driver is in the current sources.

          Here’s the simplified algorithm how to compile RT kernel:
          1. Set up the arm-linux-gnueabihf tools
          2. Download kernel source code from Raspberry Foundation Github
          3. Apply rt patch of corresponding version to the kernel source
          4. Enable Full RT preemption option by going through ‘make menuconfig’
          5. Compile kernel and modules
          6. Copy kernel and modules to the SD card
          7. Specify your new rt kernel in /boot/config.txt (if you named it differently) like: kernel=kernel-rt.img
          8. Maybe play with command line options in /boot/cmdline.txt: sdhci_bcm2708.enable_llm=0 dwc_otg.speed=1

          Feel free to ask any questions, I’ll try to help as much as I can.

          1. Jonathan says:

            Mikhail,

            Thank you for your very detailed instructions. I managed to get the kernel cross compiled using the 3.12.24 raspberry pi kernel sources and the corresponding RT patch (3.12.24-rt38). Configured it with make menuconfig, and then installed it to my sd card.

            Unfortunately, i’ve run into some issues getting it to boot. After I added in those two options you suggested, I get a kernel panic on boot:
            VFS: Mounted root (ext4 filesystem) on device
            devtmpfs: mounted
            Freeing unused kernel memory 144K
            usb 1-1.1: new high-speed USB device number 3 using dwc_otg
            Kernel panic – not syncing: No init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
            CPU: 0 PID: 1 Comm: swapper Not tainted 3.12.26-rt40+

            After some initial googling, it sounds like either a filesystem issue, or that my disk is corrupted. I’ll reimage the sd card and then reinstall, and let you know how that goes. Does this look like anything you’d recognize?

          2. Mikhail Avkhimenia says:

            Have you tried booting without those options? They were required for previous Raspberry models and may not be needed on B+.

          3. Jonathan says:

            Mikhail,

            It seems that it is working now. I didn’t get my USB keyboard working with the dwg_otc.speed=1 setting.. It appears that all that was required was the other one.

            I’d gladly share the kernel image with you (and the system image too). I just need to find a good place to upload it. When I have time tonight I may throw it on dropbox, unless you have another suggestion.

            Thanks for your help!

          4. Great work!
            We’d be very grateful for the image. Dropbox would be totally fine and I can upload it to our website to make it available for everyone. Also, it may be helpful for someone if you include sources of patched kernel, so people can compile kernel modules using them.

          5. One of our team members was working in parallel with you on the same task and also made an image of new raspbian with rt-kernel. We’ll upload it soon.

          6. Jonathan says:

            Awesome! I was having issues with Dropbox yesterday when I was trying to upload, so that’s good. Let me know if you still would like me to send anything. The process was pretty easy once I figured out what version of the patch I needed.

  7. Aaron says:

    Thanks for your prompt reply. I look forward to the update.

  8. Leo Chuah says:

    Hi, I’m rather new to raspberry pi handling real time, I have loaded this image to my raspberry pi. However, I have no idea as to how is programming a real-time C file done. Is it any different from the polling method? What program do you use to handle real-time programming?

    Also, do you know of any decent tutorials I can base on to learn how to program with a real-time based C file based on this kernel?

    A list of the functions used for real -time programming and a hello world tutorial that involves multi-threading, semaphores and priorities would be great if there is any that you know of.

    Really sorry for the bunch of questions which may seem like spoon feeding.

    1. Leo, there’s an excellent wiki devoted to the RT Linux – you can find a lot of information on rt.wiki.kernel.org.

      Also, read about chrt command here, it may come helpful.

  9. Max says:

    Hello, is there a chance to download the systemimage or rt kernel for the raspberry B+ model. I would like to try it out on the new model. Thank you!

    1. Jonathan says:

      I have a patched rt kernel I could send you, but won’t be able to do so until I get home tonight. Not sure when Mikhail will be updating the link.

      1. Jonathan says:

        Max, see the links I posted in the discussion below.

        1. Hello, could you give me and link to img for B +?

          1. Hello Konstantin,
            I’ll upload the new image soon, meanwhile you can grab the image provided by Jonathan Seawright in the comments.

  10. Paulo Alves says:

    Hi,
    Where did you get the script to generate that “fancy” chart?
    Regards
    Paulo Alves

    1. Hi Paulo,
      cyclictest utility can automatically generate histogram chart data for gnuplot which we used.

  11. Jonathan says:

    For anyone interested in trying out my raspberry pi B+ image with rt_preempt enabled, here’s some links:

    Unlike the files Mikhail shared, I haven’t done much configuration beyond enabling RT_PREEMPT and turning off frequency scaling / power management. If you would like to configure more, Ive provided the patched source tree here: https://www.dropbox.com/s/k6eiu92nxahaxpx/rt_kernel_sources_CLEAN.tar.gz?dl=0

    Compiled kernel image: https://dl.dropboxusercontent.com/u/47817073/kernel-rt.img
    To use this image, you will need to update the firmware on the SD card. Also, add “sdhci_bcm2708.enable_llm=0” to /boot/cmdline.txt

    Full system image: https://dl.dropboxusercontent.com/u/47817073/rt_raspbian_B%2B.tar.gz

  12. Hey
    I do have a question, whats the difference between RPI-with xenomai and RPI-RT ?? Both are used for real time applications? I need a Raspberry pi configured with real time capabilities to work with my drone, The control algorithm runs pretty fast, Basically I want a pi with low latency real time capabilities, so please suggest me if should go for this image or the ones complied with xenomai kernel.

  13. Its me again, I am getting build errors, please walk me through…
    I am using the stable raspberry pi source from :- https://github.com/raspberrypi/linux/
    1) So basically I download the files, extract, set the compiliers prefix and so on ..

    2) then I copy the .config from /proc directory from the working stable raspberry pi image (once which is provided in this website link)

    3) Apply the patch (patch-3.12.26-rt40.patch.gz) for rt kernel which is from this link : https://www.kernel.org/pub/linux/kernel/projects/rt/3.12/ just one file, I dont know what other patches are for..
    How do i patch ?? by following the reference – http://www.raspberrypi.org/documentation/linux/kernel/patching.md

    4) make ARCH=arm CROSS_COMPILE=${CCPREFIX} oldconfig
    5) make ARCH=arm CROSS_COMPILE=${CCPREFIX}

    when I build in the step 5, I get build errors , I tried without the patch it builds fine though.
    error reproduction :
    CC mm/filemap.o
    CC mm/mempool.o
    CC mm/oom_kill.o
    CC mm/fadvise.o
    CC mm/maccess.o
    CC mm/page_alloc.o
    mm/page_alloc.c: In function ‘free_pcppages_bulk’:
    mm/page_alloc.c:702:12: error: ‘src’ undeclared (first use in this function)
    mm/page_alloc.c:702:12: note: each undeclared identifier is reported only once for each function it appears in
    mm/page_alloc.c: In function ‘drain_pages’:
    mm/page_alloc.c:1295:4: error: implicit declaration of function ‘isolate_pcp_pages’ [-Werror=implicit-function-declaration]
    mm/page_alloc.c:1300:4: warning: passing argument 3 of ‘free_pcppages_bulk’ from incompatible pointer type [enabled by default]
    mm/page_alloc.c:674:13: note: expected ‘struct per_cpu_pages *’ but argument is of type ‘struct list_head *’
    mm/page_alloc.c: In function ‘free_hot_cold_page’:
    mm/page_alloc.c:1445:3: warning: passing argument 3 of ‘free_pcppages_bulk’ from incompatible pointer type [enabled by default]
    mm/page_alloc.c:674:13: note: expected ‘struct per_cpu_pages *’ but argument is of type ‘struct list_head *’
    cc1: some warnings being treated as errors
    make[1]: *** [mm/page_alloc.o] Error 1
    make: *** [mm] Error 2

    I am Java programmer, not much into c++, but I do see wrong, I dont want to go through the more details since i work on other domain,
    I just wanted to ask, when you build the new kernel for B+ Board, I suppose you followed the same steps as i did since the stable version is 3.12.y, how did you manage to build without errors?? I am building it in the wrong way by not applying other patches like (patches-3.12.26-rt40.tar.gz) I dont know why this is there for anyway, since as i refered most of the web pages they only apply one patch for rt which is patch-3.12.26-rt40.patch.gz (different version may be)..
    May be my question is quite simple, but I am really struggling with this, I also wanna ask one more, is it possible also to apply xenomai path along with this real time patch and have xenomai core kernel along with rpi-rt kernal?? Since xenomai has very good promising performance when it comes to real time applications

    1. Jonathan says:

      If I remember correctly, the rt40 patch is too new. The key to getting it compiled was to ensure that you get the proper version for the rpi variant of the kernel. You also might check that you’re in the correct git branch (git branch -a to list, git checkout to change)

      I don’t remember which I used – I’m away from the computer so I can’t check. You could write a script to try all the versions and then see which one works 🙂

      I think the xenomai kernel plus rt kernel would work… I have not tried it though. One thing, is to consider what peripherals (if any) you need. I don’t think xenomai supports USB 2.0, only 1.0 speeds if at all. I don’t know if it’s possible to use the Linux level driver to get 2.0.

    2. EMLID says:

      It is possible to use Xenomai with RT kernel. New Xenomai comes in two version – Cobalt and Mercury, one of which is an evolutoin of classic Xenomai with co-kernel and the other relies on the RT capabilities of Linux with the RT patch.
      Also, Xenomai project’s website has been recently updates and provides a lot of useful information, check it here:
      http://xenomai.org/start-here/

  14. moony says:

    Hi, I just need an additional kernel module. There is no .config file in the kernel source. When I do “make oldconfig” using the /proc/config.gz from the running system, I still get questions, is this normal?

    1. EMLID says:

      It should work okay, but it’s hard to tell what’s the reason of the problem.
      Are you trying to recompile the whole kernel with the new module or are you trying to compile the module separately?

  15. Jeff Lee says:

    Is it possible to run igh Ethercat Master on your Kernel? I tried to install it but i get “error No configured linux kernel sources” Am I telling it to look in the wrong place?

    1. Hello Jeff,
      Sorry, I am not familiar with EtherCAT, but the kernel sources are available in /usr/src/ and probably you’d need to specify the path to them somehow – try to check a couple of tutorials showing how to compile kernel modules on Raspbian, maybe they could be helpful on that issue.

  16. Jeff Lee says:

    Thanks Mikhail,
    I am new to the world of compiling Linux kernels and modules but I’ll let you know if I have any success.

    1. Thanks, I’ve added the link to the download section.

  17. Hi Guys,

    Is this new kernel (from Nov) based on the same raspbian kernel (3.12.26 i believe?) version or newer…?

    thx

    hugo

    1. Hello Hugo,
      Yes, the kernel is the same as in the previous image, but a couple of other things were added\updated.

  18. ok i downloaded it, still based on rasbian 3.12.26 right?

    hugo

  19. Mark says:

    Hi,

    is it possible to activate HIDRAW? That would be great!!

  20. Dennis De Matteis says:

    This Kernel is good work! I am currently working with simulink + raspberry pi and wanted to try a real time kernel. So i downloaded your image, extracted the kernel + modules, copied them on my simulink image and the kernel boots perfectly 🙂

    I can also verify your cyclictest results. With my old 3.12.28+ non rt kernel i had ~800us delay. Now with your 3.12.36-rt50 kernel it is not higher then ~80us.

    Still i am experiencing some trouble when i execute simulink code. Sometimes the raspberry pi seems to have “hick-ups”. If i toggle a GPIO Pin with 1000Hz sometimes a delay of ~5ms appears. I hoped this issue would be solved by your RT Kernel but sadly it still happens (even though i think got a bit better). Do you guys have any idea what could cause those delays? Shouldn’t cyclictest detect them?

    Regards,

    Dennis

    1. Hello Dennis,
      In order to get reliable 1000Hz timing you also need to lock thread’s memory and switch the scheduler to SCHED_FIFO(or RR).
      Like it’s done here – https://github.com/emlid/ardupilot/blob/master/libraries/AP_HAL_Linux/Scheduler.cpp
      If you can add these tunings to the code generated by Simulink and it should work.
      You also can switch the scheduler of a running binary by using “chrt -f -p 20 PID” command, where PID represents your process.

  21. Dennis De Matteis says:

    Hi Mikhail,

    thanks for your hints! I applied “chrt -fap 20 2914” to the running process and it already helped a lot! I’ll search for a way to execute it automatically right after starting the code. Still I have no clue how i could lock the threads memory, especially in combination with simulink. What exactly needs to be locked? Thread data such that it will not be swapped to disk? Guess I have to go a bit deeper into your given example 😉

    Regards,

    Dennis

    1. Glad to help!
      Process memory needs to be locked in order it does not get moved causing delays.
      It’s important to remember that real-time kernel does not make everything real-time by itself, only when the code is written appropriately.

  22. HollaDieWaldfee says:

    Will you publish a kernel version for the new RPi 2 B too?

Leave a Reply

Your email address will not be published. Required fields are marked *