What are Real-Time Constraints in Robotics?

Real-time constraints in robotics is often a difficult-to-understand subject at first. What does real-time really mean?

Most robotic systems are subject to real-time constraints. Not respecting those constraints when developing a robot, or a robotic application, might result in different kinds of issues. In the worst case, the application will completely fail to perform correctly. In the best case, the performance of the overall system will be lowered.

That’s why you must be aware of what are real-time constraints in robotics. More globally, real-time constraints are not really specific to robotics. They can be applied to almost anything.

In this post, I’ll explain to you – through some analogies and real life examples:

• What are real-time constraints
• What are the different kinds of real-time constraints
• Where to care about those constraints when you develop a robotics application
• A special focus on Arduino and Raspberry Pi when it comes to real-time constraints

>> Here is a video version of this tutorial, as an additional resource:

After watching the video, subscribe to the Robotics Back-End Youtube channel so you don’t miss the next tutorials!

What we commonly refer to when speaking about real time

Before diving into the explanations, I first want to talk about the common thinking behind real-time constraints. “Real-time” is a word that’s used quite often, and most of the time with a different meaning. Let’s clarify things here.

Usually, when we hear “real time”, we think of :

• An online chat (Messenger, Slack, Discord, …), where the message you type is directly visible by everyone in the chat room.
• News in real-time. You get a notification on your phone as soon as there is an important news in an area where you don’t want to miss anything.
• An application on your phone showing the current temperature in your home.

Mostly, when speaking about real-time, we speak about the “feeling” of seeing things simultaneously happen.

You type a message and your friends receive it. Then someone else answers and you receive the message. It feels like you’re talking in real-time with the other person.

With the temperature sensor, it feels like you’re seeing what’s happening in real-time in your home. Just as if you were in your home and watching the temperature sensor directly.

Well, real-time constraints are a little bit more complex than that. The feeling of real-time is what we can easily perceive, but it does not make a system becoming “real-time” or not.

What real-time really means

A real life analogy

First of all, real-time constraints are not specific to robotics and programming. The concept can be applied to almost anything we know.

A real time constraint is associated with a deadline. When you apply a real-time constraint to a system, it means that the system must respect certain rules and deadlines, in order to be executed successfully.

Let’s take a very basic example, which has nothing to do with robotics.

Imagine you’re given a challenge. This challenge is simply to run for 100 meters.

So, that’s the system. Now, to this challenge I add a real-time constraint. I say : you must finish the sprint under 40 seconds in order to successfully complete the challenge. If you finish after 61 seconds, then you’ve failed.

That’s pretty much it.

Note that I didn’t specify whether you should run fast or slow. Also, as you can see, running 100 meters in 40 seconds is really not hard to do. Real-time doesn’t mean that you should apply strong performance (more on that later in this post). You just need to respect the deadline.

For those specific conditions, you can be quite confident that you’ll always respect the deadline, and thus you can say that the real-time constraints are satisfied.

Let’s change the deadline. I now ask you to do the same thing, but the deadline is 20 seconds (I just took an arbitrary number, I didn’t really do the test myself). Meeting the deadline seems doable, but maybe not every time. If you need to repeat the challenge 100 times, you might start to feel tired and you’ll slow down. Or simply, your physical condition and/or preparation is not sufficient to complete the challenge even a few times.

Let’s come back to the 40 seconds deadline. Now I add one more constraint: you should not take less than 30 seconds to complete the challenge. Simply put, if you run too slow or too fast, you fail. The point is to arrive on time, not before, and not after.

Well, as you can see, I’ve just discussed real-time constraints without a single word related to programming or robotics. Because this concept is applicable to anything, not just technology and science.

Real time constraints in robotics

Real-time constraints are really important in robotics.

Imagine that you’re designing and optimizing a production line, including many robotic arms, conveyors, cameras, and people.

Safety concerns first: you need to make sure that all the robotic systems stop before a human can be hurt. If a human is hurt, it’s really too late and the deadline for computing and stopping the machines is totally missed.

Then, production concerns: imagine some of the cameras are used to sort objects, which are moving on a conveyor. A robotic arm will pick objects depending on the camera decision. If the camera response is coming after an object have passed under the robotic arm, then it’s too late. It’s certainly not as critical as hurting a human, but still, the task has failed.

There is also a notion of precedence of the various real-time tasks. In this production example, we have identified 2 constraints: the system must react fast enough not to hurt humans, and fast enough to avoid missing some objects to pick by the robotic arm.

But what is supposed to happen when those 2 constraints are applied at the same time? That’s where the precedence takes place. In the global system, the constraint not to hurt humans will be given a higher priority.

So, if at some point, a human is walking in a dangerous zone, and the camera is late on delivering the correct information, then the safety constraint will preempt any other lower-ranked constraints.

To quickly recap, a real-time system is a system that has to complete a task, or list of tasks, on time.

Note that real-time does not mean “fast”. The only thing the system should do is to arrive on time.

In real-life, to catch a train, you must arrive before the train leaves the station (obviously). But, whether you arrive 1 minute or 30 minutes before doesn’t change anything, you’ll still catch the train.

In a production line, if the camera has 50 milliseconds to detect an object maximum, then doing it in 10 milliseconds or 40 milliseconds will not change the outcome.

Different types of real-time constraints

Missing a deadline for a real-time task does not necessarily have the same impact and consequences.

There are 3 main types of real-time constraints, depending on your application and the results you expect: hard, firm, and soft.

For each kind of constraint, I’ll show you the impact on the first analogy we used before. To get the exact same system, let’s consider that you have to run 100 meters in 30 seconds, and you repeat this task 100 times. There is a score system that we’ll modify in each kind of constraints, and of course the goal is to make the highest score.

Hard real-time constraints

This one is the most strict. For hard-real time constraints, a failure to meet only one deadline means that the system has critically failed. There is absolutely no flexibility.

For the real-life analogy, let’s say you run under 40 seconds for the first 73 tests, and on the 74th test, it takes you 42 seconds. No matter what score you got before, or the score you could get from the 75th to the 100th test, you’ve failed, and you’re disqualified. The overall score is zero and you can go home directly.

Usually, hard real-time constraints are applied to critical systems, such as:

• A rocket in space going to the moon: what if one of the propulsor is not working on time?
• Air traffic control: it’s simply not imaginable that 2 planes could collide, and such an event would be a dramatic failure of the system
• Autonomous car control
• Nuclear reactors
• Anti-lock brakes (ABS): when you have to stop in an emergency situation, you can’t expect the ABS to work only 99.99% of the time. It needs to work 100% of the time.

Firm real-time constraints

With firm real-time constraints, missing a deadline is not fatal to the system. However, the produced result has zero value.

Let’s come back to our running challenge. Let’s say you get 10 points per challenge completed. The max score will be 1000 if you complete all 100 challenges. If, for one sprint, you take more than 40 seconds, then you’ll simply have 0 point for this sprint. Taking 41 seconds or 57 seconds will produce the same result. But, you can still run all 100 sprints and get a final score at the end.

Usually, firm real-time constraints implies a loss of productivity, or revenue:

• Some production line setups. Example: if an object is missed by a camera, it will be thrown away.
• Storm forecast system: if a storm is detected only after it has started, then the result is useless. However, this is not critical to the system, and the storm forecast can continue to work.

Soft real-time constraints

Missing a deadline with soft real-time constraints is not fatal to the system, and the result can still be valuable, to a certain point.

Running challenge again: you still get 10 points per sprint completed under 40 seconds, so you can score 1000 points maximum with the 100 sprints. But now, there is a new rule for the score of each individual sprint. For each additional second after 40 seconds, your score is lowered by 1. So, if you take 47 seconds, you only get 3 points. And, after 50 seconds, you only get 1 points for at least running 100 meters, no matter how long you took. The important thing to see here is that you still get some points even if you missed the deadline.

For soft real-time constraints, each missed deadline can degrade the overall system performance. This can often be associated to what you know as a quality of service:

• Video streaming: Some pixels are sometimes not displayed correctly, or the bandwidth can be slow for a minute. The video will still play, but with a lower resolution or quality.
• Video games: You might experience some lags due to your computer being not powerful enough for some animations. You can still continue to play the video game and finish it, but your overall experience is degraded.
• Web page loading time: the longer a web page takes to load, the more viewers you loose.
• Some production line setups. Example: even if a camera is late to give the coordinate of an object in a conveyor to a robotic arm, the arm will still be able to pick the object because the conveyor will stop. In this case, no object is missed, but the total time to complete the batch of objects is longer.

As a robot developer, when do you need to care about real-time constraints

If you develop a robot, it’s important for you to know when you need to think about those constraints – and when it’s just not worth it.

Low level code

Almost everything which is low level and close to hardware is subject to real-time constraints.

For example, a firmware that controls a servo motor. The firmware needs to give an appropriate command to the motor at a certain frequency so the movement is smooth. If some commands are missed, the axis where the motor is attached may have some unwanted jerk (quick changes in acceleration). Having jerk on a robot may lead to premature wear on some parts, or worse, failure of the system to correctly complete a task.

Let’s say you have a robot for which the main task is to do some laser engraving, following a predefined pattern. If the robot has some unwanted jerk, the laser tool won’t be able to follow the pattern correctly, and you can just throw the result away.

Decision algorithms

In some situations, your robot will have to make important decisions: when to stop and where/when to turn for an autonomous car, how to re-plan a trajectory for a robotic arm, etc. The decision process here have real-time constraints. You need to make sure that the robot decides what to do before the decision becomes obsolete, and it’s too late to avoid security hazards, breaks in the system, or failure to complete a given task.

Synchronization between devices

Usually, you don’t use only one robot, or one motor. Your application is often a set of multiple robotic components interacting with each other.

In a traditional assembly line, you’ll have to make all devices communicate together, synchronously: PLCs (Programmable Logic Controller), robots, conveyor belts, …

If one PLC is responsible for managing when a robot A should move a part, and when robot B should take the part, then a failure of the PLC to respond fast enough will lead to a failure to complete the task.

In a simple 2 wheels mobile base controlled with 2 servo motors and one Arduino board, real-time synchronization of the 2 motors is very important. If the 2 motors are not correctly synchronized, you won’t be able to make the robot go straight!

Are Arduino and Raspberry Pi real-time systems?

Microcontrollers (and Arduino)

An Arduino board contains a microcontroller (ATmega328P for Arduino Uno). A microcontroller has no real OS. Basically, it will execute the program you wrote, line by line.

A microcontroller can be considered as a real-time system. You can be sure that no executed task will be preempted by an operating system allocating more or less resources for some applications.

So, using a microcontroller (Arduino board if you’re starting) for low level code is quite appropriate. You can control motors, communicate with other devices, trigger some PWMs, read values from sensors, and all that with real-time constraints.

Raspberry Pi board

A Raspberry Pi is basically a normal computer, but on a credit card size PCB. You can install some Windows, Ubuntu, Debian (Raspbian) distributions and run your Raspberry Pi as you would run your laptop.

However, know that all those operating systems are not real-time. They manage the resources allocated to all the different programs running simultaneously, in a way that allows all applications to run, and new applications to start not too slowly. For example, if you have installed Firefox on your Ubuntu OS, then when you start it, other applications may slow down or even freeze for a while, so Firefox can correctly start.

So it means that when using a non real-time operating system, you can’t be sure that your application will always get the necessary resources needed to respect the timing constraints you have.

A solution is to use a real-time operating system, for example RTOS.

Conclusion: it’s all about the balance between real-time and non real-time

All your application does not necessarily need to respect real-time constraints. Writing code that respects those constraints if often harder and takes more development time.

The best solution when designing and implementing a robotics application, is first to check what areas of your application need to respect real-time constraints, and also which ones don’t need to do that.

Sort those constraints by their important: hard, firm, or soft.

Then, analyze what hardware, operating system, software practice, you need to use in order to respect those constraints.

Finally, you’ll have a system containing both real-time safe zones and non real-time zones.

You’ll need to ensure that the communication between those different zones can be made in order to respect the constraints on the safe zones.

All in all, real-time constraints are really important to take into account when you develop a robot software.