[Tweets]

11/06/2018 5:20pm
RT @AndyRidgway1: They've been a fair few months in the making but we - that's @SciCommsUWE - have just launched a new range of #scicomm tr…
08/06/2018 11:19am
MIT scientists created a “psychopath” AI by feeding it violent content from Reddit. A very interesting experiment.… [LINK]
07/06/2018 7:36pm
Excited that our paper on #information exchange #design #patterns for #robot #swarm #foraging is now published in… [LINK]
06/06/2018 5:04pm
I suppose following the horrible news of @Microsoft taking over @github, this is what the @Bitbucket landing page l… [LINK]

[V-REP, Gazebo or ARGoS? A robot simulators comparison]

Multi-agent systems Robots C++ AI Added on 17/01/2018

V-REP, Gazebo and ARGoS logos

In agent-based modelling for robotics, it is important to pick a simulation environment that best suits your project. Do you go for speed, model detail, or for a rich set of features? Let’s have a look at three commonly used open-source simulators for robotics: V-REP, Gazebo and ARGoS, to find out which one suits your project the best.

  • V-REP is a very feature-rich simulation environment that includes a scene and model editor, a large library of models, real-time mesh manipulation, etc. It is available under a commercial licence or for free for educational institutions.
  • ARGoS provides a lightweight alternative that is especially relevant for swarm robotics.
  • Gazebo represents a middle ground between V-REP and ARGoS. While it can utilise complex 3D meshes and physics engines, it lacks some features that V-REP has, such as mesh manipulation.


This article is based on work that I have done for the Robotics for Nuclear Environments project at the Bristol Robotics Laboratory (BRL). It has been published with the BRL permission.

A detailed look at the simulator characteristics

The built-in capabilities, model library, programming methods and user interface of the simulators are compared below. Characteristics with a relative positive impact are marked in green and those with a negative impact in red.

V-REP Gazebo ARGoS
Available for MacOS, Linux and Windows. Binary packages are available for all platforms. Available for MacOS, Linux and Windows. A binary package is only available for Linux Debian. Gazebo is installed via the command line using third-party package managers on other systems. Available for MacOS and Linux. Binary packages are available for Linux. On MacOS, ARGoS is installed via the command line using a third-party package manager.
Built-in capabilities
Default physics engines include: Bullet 2.78, Bullet 2.83, ODE, Vortex and Newton. Only the ODE physics engine is available by default. It is, however, possible to build Gazebo from source with a different physics engine. A 2D and a 3D custom-built physics engines with very limited capabilities are available by default.
Includes a code and a scene editor. Includes a code and a scene editor. Includes a Lua script editor but no scene editor.
Scene objects can be fully interacted with (e.g., moved or added) by the user during simulation. The world returns to its original state when the simulation is reset. Scene objects can be fully interacted with (e.g., moved or added) by the user during simulation. The world does not return to its original state when the simulation is reset. Scene objects can be moved by the user during simulation.
Meshes can be manipulated (e.g., cut) by robots in real time. No mesh manipulation is available. No mesh manipulation is available.
Outputs include video, custom data plots and text files. Outputs include simulation log files, video frames as pictures and text files. Outputs include video frames as pictures and text files.
Includes particle systems. No particle systems are available. No particle systems are available.
Robot and other models
Provides a large variety of robots, including bi-pedal, hexapod, wheeled, flying and snake-like robots. Also provides a large number of robot actuators and sensors. A less diverse library of default robots, that mostly includes wheeled and flying robots. Third-party robot models are available, but their documentation is often poor. A fairly small library of robots, only including the e-puck, eye-bot, Kilobot, marXbot, and Spiri robots.
The default models are very detailed and therefore appropriate for high-precision simulations. It is possible to simplify the models in V-REP. The default models are fairly simple and are therefore more appropriate for computationally complex simulations. The default models are fairly simple and are therefore more appropriate for computationally complex simulations.
Meshes are imported as collections of sub-components. It is therefore possible manipulate individual parts of an imported model and to change their textures, materials and other properties. Meshes are imported as single objects. Models that contain multiple sub-components have to be assembled in Gazebo from multiple DAE files, each corresponding to one sub-component. Mesh importing is not available. Object representations are coded using OpenGL.
It is possible to simplify, split and combine meshes. This makes it possible to optimise the triangle count of imported models and to manipulate meshes (e.g., to cut them) with robot actuators. Imported meshes cannot be changed. A model therefore has to be optimised in a third-party 3D modeling software. This may be difficult during iterative development. N/A
Programming methods
A scene is saved in a special V-REP format. All scene editing therefore has to be done using the V-REP interface. A scene is saved as an XML file. This makes is possible to e.g. create a bash script that changes the scene and then runs a simulation. A scene is saved as an XML file. This makes is possible to e.g. create a bash script that changes the scene and then runs a simulation.
There are various options for implementing functionality, including scripts attached to robots, plug-ins, ROS nodes or separate programs that connect to V-REP via the RemoteAPI. Functionality can be implemented either as compiled C++ plug-ins or via ROS programs. Lack of scripting makes it difficult to run quick tests with ad-hoc solutions. Robots can be programmed either through Lua scripts or in C++.
Scripts can be included in robot models and are often used to desscribe the models and their capabilities. It is difficult to recognise how a third-party robot model works or what plug-ins it uses. The plug-in list is only available in the Model Editor. Some documentation of the robots is provided in ARGoS, but most of how a robot works needs to be deducted from code examples.
"CustomUI" API, based on QT, is used to create custom interfaces. Custom UI controllers can be attached to individual robots. For example, it is possible to display a robot's camera output when the robot is clicked on. Custom interfaces can be created as plug-ins by using the default QT API. The interfaces can only be attached to the whole scene and not to individual robots. Custom interfaces can be created in C++ by subclassing an ARGoS API class. The interfaces can be attached to the whole scene or to invididual robots.
All scripts and plug-ins provided with the default robot models and example scenes worked. Many plug-ins provided with the default robot models did not work. Some on-line examples were difficult to run due to a large number of dependencies and differences in ROS versions. A few examples are provided on the ARGoS website, all of which worked.
Good API documentation, a large library of tutorials and code examples and a large user community are available. Regular updates have been provided since 2013. A fairly comprehensive documentation, step-by-step tutorials and a large user community are available. Gazebo is likely to be supported in the future, since a development road map is available on the website. Good documentation, but a small user community are available. Development has not been regular. Updates can be requested but can take a long time to get developed.
User interface (UI)
No freezing issues with the interface were experienced. The interface froze a number of times and the program, and sometimes the computer, had to be restarted. No freezing issues with the interface were experienced.
All functionality is fairly intuitive and follows general conventions known from similar applications. The UI usability is relatively low. For example, the top application tool bar sometimes disappears, it is not possible to copy and paste multiple objects, or to save a scene into the same file after making changes to it. The UI is very limited, but all functionality is fairly intuitive and follows general conventions known from similar applications.
The model library is distributed with V-REP and it is thus always available regardless of Internet connectivity. The model library is not distributed with Gazebo, and it is instead available on-line. On multiple occasions, the library could not be accessed because Gazebo could not connect to its server, even though the computer was connected to the Internet. The robot models are distributed with ARGoS and it is thus always available regardless of Internet connectivity.
The model library is organised into folders based on model category. User can create their own folders for their own models. The model library is a long list of models and particular model types (e.g., robots) can be difficult to find. There is no model library. Instead, different robot types are natively a part of ARGoS.

A detailed look at performance

There were two types of test:

  • GUI benchmark involved running a simulator along with its user interface, and attempting to run a 1-minute long simulation in real-time.
  • Headless benchmark involved running a simulator from the command line, without its user interface, and attempting to run a 5-minute long simulation as quickly as possible.


There were two types of environment:

  • Small scene, where robots were put on a large 2D plane
  • Large scene (screenshot below), where 5,200 boxes (total of 41,600 vertices) were randomly generated, with robots placed randomly between them


Large scene in ARGoS

Robot models were selected from a library of models available in each simulator, so that a sensible similarity, in terms of robot geometry and controller capabilities, was achieved. In V-REP, the original e-puck model was simplified in order to decrease its vertex count.

E-puck   Pioneer 3AT   Eye-bot
A V-REP screenshot of the simplified (left) and original (right) e-puck models   A Gazebo screenshot of the Pioneer 3AT robot model   An ARGoS screenshot of the eye-bot model



Three performance metrics were used:

  • R: Real-time factor, calculated as R = simulated time / real time. When R = 1, the simulation speed corresponded to real time. When R < 1, the simulation ran slower than real time
  • C: CPU usage. A usage over 100% indicates that multiple processor cores were used
  • M: Memory (RAM) usage.


All tests were performed in a 64-bit Ubuntu Linux 16.04 environment running natively on a MacBook Pro with 4x Intel Core i7 2.2Gz, 8GB RAM and Intel HD Graphics 6000.

Because of long computation times, some experiments were not feasible to run. These cases are marked as “Not feasible” in the tables. Also notice that two values for C and M are reported for Gazebo GUI experiments, corresponding to usage of gzclient (the user interface) and gzserver (the simulator), respectively.

Simulator performance in the GUI mode, with the best (green) and the worst (red) performance indicated:

  V-REP +
Bullet 2.78
V-REP +
ODE
Gazebo +
ODE
ARGoS +
PointMass3D
1 robot +
Small scene
R ≥ 1
C = 180%
M = 235 MB
R ≥ 1
C = 190%
M = 225 MB
R ≥ 1
C = 104+9%
M = 225+58 MB
R ≥ 1
C = 7%
M = 85 MB
5 robots +
Small scene
R = 0.52
C = 395%
M = 380 MB
R = 0.37
C = 395%
M = 360 MB
R ≥ 1
C = 105+19%
M = 305+58 MB
R ≥ 1
C = 10%
M = 88 MB
10 robots +
Small scene
R = 0.11
C = 400%
M = 536 MB
R =0.099
C = 400%
M = 530 MB
R ≥ 1
C = 105+30%
M = 402+58 MB
R ≥ 1
C = 13%
M = 89 MB
50 robots +
Small scene
Not feasible Not feasible R = 0.87
C = 109+105%
M = 1410+358 MB
R = 0.9
C = 103%
M = 93 MB
1 robot +
Large scene
R = 0.96
C = 205%
M = 235 MB
R = 0.53
C = 200%
M = 225 MB
R ≥ 1
C = 104+10%
M = 264+58 MB
R ≥ 1
C = 32%
M = 90 MB
5 robots +
Large scene
R = 0.18
C = 400%
M = 325 MB
R = 0.1
C = 400%
M = 310 MB
R ≥ 1
C = 105+25%
M = 333+58 MB
R ≥ 1
C = 60%
M = 97 MB
10 robots +
Large scene
R = 0.052
C = 400%
M = 433 MB
R = 0.036
C = 400%
M = 460 MB
R ≥ 1
C = 105+40%
M = 425+58 MB
R = 0.86
C = 120%
M = 97 MB
50 robots +
Large scene
Not feasible Not feasible R = 0.57
C = 109+105%
M = 1450+426 MB
R = 0.052
C = 107%
M = 106 MB



Simulator performance in the Headless mode, with the best (green) and the worst (red) performance indicated:

  V-REP +
Bullet 2.78
V-REP +
ODE
Gazebo +
ODE
ARGoS +
PointMass3D
1 robot +
Small scene
R = 4.1
C = 200%
M = 165 MB
R = 3.12
C = 200%
M = 160 MB
R = 42.85
C = 103%
M = 107 MB
R = 300
C = 6.3%
M = 18 MB
5 robots +
Small scene
R = 0.38
C = 400%
M = 320 MB
R = 0.32
C = 400%
M = 320 MB
R = 10
C = 103%
M = 130 MB
R = 150
C = 100%
M = 20 MB
10 robots +
Small scene
R = 0.09
C = 400%
M = 470 MB
R = 0.08
C = 400%
M = 480 MB
R = 5.26
C = 103%
M = 150 MB
R = 21.42
C = 144%
M = 20 MB
50 robots +
Small scene
Not feasible Not feasible R = 1.06
C = 103%
M = 356 MB
R = 0.52
C = 103%
M = 25 MB
1 robot +
Large scene
R = 1.91
C = 200%
M = 165 MB
R = 0.58
C = 200%
M = 160 MB
R = 18.75
C = 103%
M = 174 MB
R = 15.78
C = 139%
M = 31 MB
5 robots +
Large scene
R = 0.2
C = 400%
M = 270 MB
R = 0.11
C = 400%
M = 250 MB
R = 5.88
C = 103%
M = 192 MB
R = 5.45
C = 157%
M = 45 MB
10 robots +
Large scene
Not feasible Not feasible R = 3.09
C = 103%
M = 211 MB
R = 1.59
C = 130%
M = 47 MB
50 robots +
Large scene
Not feasible Not feasible R = 0.60
C = 103%
M = 423 MB
R = 0.03
C = 105%
M = 55 MB

The verdict

V-REP logoV-REP is the most complex and the most resource-hungry of the three simulators. However, it offers a number of useful features and is therefore a superior choice for detailed and good-looking simulations that can be run on powerful machines. It uniquely includes features such as the choice of multiple physics engines, a comprehensive model library, the ability of a user to interact with the world during simulation, video recording and excellent documentation. V-REP also offers superior options for model mesh optimisation, meaning that the need for computational resources can be relatively easily decreased without using any third-party 3D modeling software. Moreover, V-REP automatically spawns new threads on multiple CPU cores and therefore utilises the full amount of CPU power when necessary. Gazebo failed to exhibit this ability and some problems were noted in ARGoS1 during the experiments performed here.

ARGoS logoOn the other hand, ARGoS is a suitable choice for swarm robotics experiments, although it provides a fairly small robot library. It trades-off robot, environment and physics complexity for superior performance. An XML-based simulation settings file is also very convenient, especially when a large variety of simulations need to be generated and run automatically. However, there are multiple important features missing from ARGoS, most notably the ability to import 3D meshes into the simulator. Without this feature, users that are not willing to spend time and effort for programming new robot models in OpenGL, have very limited choices.

Gazebo logoGazebo occupies the space between V-REP and ARGoS. While it is much closer to V-REP in terms of features, its interface and default robot models are much simpler and resemble those found in ARGoS. It is notable that Gazebo outperformed ARGoS in the larger simulation environments studied here, which suggests that it is a more suitable choice for large swarm robotics experiments. However, the usability of Gazebo is relatively poor. Firstly, while it can import 3D meshes, there are no editing options, making it difficult to alter and optimise models. Another problem is the interface that has a number of issues and fails to follow established conventions. Finally, difficulties were noted when installing dependencies for Gazebo and for many of its third-party models. While not necessarily severe by themselves, these issues together could bring a lot of frustrations and set-backs during a project, especially in a large team.

Note 1: ARGoS does support multi-threading, however, it had problems with spawning threads in experiments with a large number of robots, making the CPU usage increasingly smaller with an increasing number of robots. This problem occurred in both MacOS and Linux environments.

Edits on 18/Jan/2018:
Thanks to Giovanni Beltrame and Carlo Pinciroli for alerting me to some features of ARGoS, such as the ability to interact with robots, create custom UI widgets and using Kilobots. The text has been changed to reflect this. Also, thanks to Giovanni for pointing out that multi-threading in ARGoS should work. Since it worked well for swarms of up to 50 robots, the text has been changed to reflect this. Also, the performance of ARGoS was re-tested with multi-threading on and the numbers in the results tables have been changed.

Comments

Michele Colledanchise
[05/04/2018]
Some other colleagues are stuck on the same problem with Windows 10. Moreover, the fact that OSRF marked the issue "bug" with "major" priority suggests that it could be a problem on their side. I have to say that I usually use Ubuntu for programming so there is the possibility that I made a silly mistake somewhere with the Windows libraries. However, I haven't seen anybody using Gazebo on Windows.
Lenka
[05/04/2018]


Hi Michele, thanks for your message. Yes, you are correct that for many simulations and machines, CPU usage might not be a problem. It basically becomes a problem when you want to simulate a lot of robots working together.

I have not tried Gazebo on Windows or MacOS, mostly because of the many dependencies that often need to be installed manually. I can see that you have a problem with dependencies, but it is strange that the files are physically there, yet your compiler cannot find them. Is it possible that the libraries were built on a different version of Windows and you just copied them over, so the compiler is not recognizing them as something usable?
Michele Colledanchise
[04/04/2018]
Hi Lenka,
Thank you for sharing this. This is the type of comparison I have been looking for in a while since I personally prefer V-REP over Gazebo and many people ask me why.

The CPU usage or V-REP is not necessarily a bad thing since it is using the available resources of your machine.

I have a question: Have you tried yourself to install Gazebo on MacOS or Windows? Because I tried to install gazebo in windows and it does not work (see the issue here https://bitbucket.org/osrf/gazebo/issues/2398/installation-problem-on-windows-due-to and here https://bitbucket.org/osrf/gazebo/issues/2351/gazebo-issues-on-windows-subsystem-for)

Again, Thank you

Michele

Lenka
[28/02/2018]


Thanks Jessica!
Jessica
[28/02/2018]


Thanks, Lenka. I have been looking everywhere for something like this and your comparison is the best one out there.


{Please enable JavaScript in order to post comments}

Robot swarms in action

Watch e-puck robots collect resources and bring them back to base. While the previous simulation work helped us to learn a lot about the advantages and disadvantages of communication in swarms, doing similar experiments with real robots is already revealing interesting new things.

How to compile code for e-puck robots on your computer

Compiling code natively on e-puck robots or cross-compiling on your computer can be very tedious. Luckily, there is a third option: compiling code on a virtualised robot system on your computer, then sending the program onto the robot.

pyCreeper

The main purpose of pyCreeper is to wrap tens of lines of python code, required to produce graphs that look good for a publication, into functions. It takes away your need to understand various quirks of matplotlib and gives you back ready-to-use and well-documented code.

Citations count fails to measure the impact of research

Academic career progress is often judged by so-called "h-index", that measures how much your research is being cited. I recently had a detailed look at what exactly the citations, specifically those reported by Google Scholar, amount to. Only a relatively small fraction of the reported citations corresponded to research being applied or reproduced in a meaningful way.

Designing Effective Roadmaps for Robotics Innovation

Automated factories, autonomous delivery drones, self-driving cars: these and similar technologies will soon touch every aspect of our lives. An engaging discussion about how these technologies are regulated and innovated took place at the IROS 2017 conference.

The Information-Cost-Reward framework for understanding robot swarm foraging

The Information-Cost-Reward (ICR) framework relates the way in which robots obtain and share information about where work needs to be done to the swarm’s ability to exploit that information in order to perform work efficiently in the context of a particular task and environment.