[V-REP, Gazebo or ARGoS? A Robot Simulators Comparison]

Tutorials Robots Multi-agent systems C++ AI

Date: Jan 2018

Publication: Pitonakova, L., Giuliani, M., Pipe, A., Winfield, A. (2018) Feature and performance comparison of the V-REP, Gazebo and ARGoS robot simulators. Proceedings of the 19th Towards Autonomous Robotic Systems Conference (TAROS 2018), Lecture Notes in Computer Science, vol 10965, Springer, 357-368.

Disclaimer: This article and the research paper have been written with the use of V-REP 3.5.0, Gazebo 8.1.1. and ARGoS 3.0.0-beta48. Newer versions may vary. Based on work that I have done for the Robotics for Nuclear Environments project at the Bristol Robotics Laboratory (BRL). The article has been published with permission from the BRL.

In agent-based modelling for robotics, it is important to pick a simulation environment that best suits your project. Do you go for speed, model detail, or for a rich set of features? Let’s have a look at three commonly used open-source simulators for robotics: V-REP, Gazebo and ARGoS, to find out which one suits your project the best.

• V-REP is a very feature-rich simulation environment that includes a scene and model editor, a large library of models, real-time mesh manipulation, etc. It is available under a commercial licence or for free for educational institutions.
• ARGoS provides a lightweight alternative that is especially relevant for swarm robotics.
• Gazebo represents a middle ground between V-REP and ARGoS. While it can utilise complex 3D meshes and physics engines, it lacks some features that V-REP has, such as mesh manipulation.

A detailed look at the simulator characteristics

The built-in capabilities, model library, programming methods and user interface of the simulators are compared below. Characteristics with a relative positive impact are marked in green and those with a negative impact in red.

V-REP	Gazebo	ARGoS
Available for MacOS, Linux and Windows. Binary packages are available for all platforms.	Available for MacOS, Linux and Windows. A binary package is only available for Linux Debian. Gazebo is installed via the command line using third-party package managers on other systems.	Available for MacOS and Linux. Binary packages are available for Linux. On MacOS, ARGoS is installed via the command line using a third-party package manager.
Built-in capabilities
Default physics engines include: Bullet 2.78, Bullet 2.83, ODE, Vortex and Newton.	Only the ODE physics engine is available by default. It is, however, possible to build Gazebo from source with a different physics engine.	A 2D and a 3D custom-built physics engines with very limited capabilities are available by default.
Includes a code and a scene editor.	Includes a code and a scene editor.	Includes a Lua script editor but no scene editor.
Meshes can be manipulated (e.g., cut) by robots in real time.	No mesh manipulation is available.	No mesh manipulation is available.
Scene objects can be fully interacted with (e.g., moved or added) by the user during simulation. The world returns to its original state when the simulation is reset.	Scene objects can be fully interacted with (e.g., moved or added) by the user during simulation. The world does not return to its original state when the simulation is reset.	Scene objects can be moved by the user during simulation.
Outputs include video, custom data plots and text files.	Outputs include simulation log files, video frames as pictures and text files.	Outputs include video frames as pictures and text files.
Includes particle systems.	No particle systems are available.	No particle systems are available.
Robot and other models
Provides a large variety of robots, including bi-pedal, hexapod, wheeled, flying and snake-like robots. Also provides a large number of robot actuators and sensors.	A less diverse library of default robots, that mostly includes wheeled and flying robots. Third-party robot models are available, but their documentation is often poor.	A fairly small library of robots, only including the e-puck, eye-bot, Kilobot, marXbot, and Spiri robots.
The default models are very detailed and therefore appropriate for high-precision simulations. It is possible to simplify the models in V-REP.	The default models are fairly simple and are therefore more appropriate for computationally complex simulations.	The default models are fairly simple and are therefore more appropriate for computationally complex simulations.
Meshes are imported as collections of sub-components. It is therefore possible manipulate individual parts of an imported model and to change their textures, materials and other properties.	Meshes are imported as single objects. Models that contain multiple sub-components have to be assembled in Gazebo from multiple DAE files, each corresponding to one sub-component.	Mesh importing is not available. Object representations are coded using OpenGL.
It is possible to simplify, split and combine meshes. This makes it possible to optimise the triangle count of imported models and to manipulate meshes (e.g., to cut them) with robot actuators.	Imported meshes cannot be changed. A model therefore has to be optimised in a third-party 3D modeling software. This may be difficult during iterative development.	N/A
Programming methods
A scene is saved in a special V-REP format. All scene editing therefore has to be done using the V-REP interface.	A scene is saved as an XML file. This makes is possible to e.g. create a bash script that changes the scene and then runs a simulation.	A scene is saved as an XML file. This makes is possible to e.g. create a bash script that changes the scene and then runs a simulation.
There are various options for prorgramming functionality, including scripts attached to robots, plug-ins, ROS nodes or separate programs that connect to V-REP via the RemoteAPI.	Functionality can be programmed either as compiled C++ plug-ins or via ROS programs. Lack of scripting makes it difficult to run quick tests with ad-hoc solutions.	Robots can be programmed either through Lua scripts or in C++.
Scripts can be included in robot models and are often used to desscribe the models and their capabilities.	It is difficult to recognise how a third-party robot model works or what plug-ins it uses. The plug-in list is only available in the Model Editor.	Some documentation of the robots is provided in ARGoS, but most of how a robot works needs to be deducted from code examples.
"CustomUI" API, based on QT, is used to create custom interfaces. Custom UI controllers can be attached to individual robots. For example, it is possible to display a robot's camera output when the robot is clicked on.	Custom interfaces can be created as plug-ins by using the default QT API. The interfaces can only be attached to the whole scene and not to individual robots.	Custom interfaces can be created in C++ by subclassing an ARGoS API class. The interfaces can be attached to the whole scene or to invididual robots.
All scripts and plug-ins provided with the default robot models and example scenes worked.	Many plug-ins provided with the default robot models did not work. Some on-line examples were difficult to run due to a large number of dependencies and differences in ROS versions.	A few examples are provided on the ARGoS website, all of which worked.
Good API documentation, a large library of tutorials and code examples and a large user community are available. Regular updates have been provided since 2013.	A fairly comprehensive documentation, step-by-step tutorials and a large user community are available. Gazebo is likely to be supported in the future, since a development road map is available on the website.	Good documentation, but a small user community are available. Development has not been regular. Updates can be requested but can take a long time to get developed.
User interface (UI)
No freezing issues with the interface were experienced.	The interface froze a number of times and the program, and sometimes the computer, had to be restarted. This occurred, e.g., when editing robot models, starting or stopping the simulation, and in other instances.	No freezing issues with the interface were experienced.
All functionality is fairly intuitive and follows general conventions known from similar applications.	The UI usability is relatively low. For example, the top application tool bar sometimes disappears, it is not possible to copy and paste multiple objects, or to save a scene into the same file after making changes to it.	The UI is very limited, but all functionality is fairly intuitive and follows general conventions known from similar applications.
The model library is distributed with V-REP and it is thus always available regardless of Internet connectivity.	The model library is not distributed with Gazebo, and it is instead available on-line. On multiple occasions, the library could not be accessed because Gazebo could not connect to its server, even though the computer was connected to the Internet.	The robot models are distributed with ARGoS and it is thus always available regardless of Internet connectivity.
The model library is organised into folders based on model category. User can create their own folders for their own models.	The model library is a long list of models and particular model types (e.g., robots) can be difficult to find.	There is no model library. Instead, different robot types are natively a part of ARGoS.

A detailed look at performance

There were two types of test:

• GUI benchmark involved running a simulator along with its user interface, and attempting to run a 1-minute long simulation in real-time.
• Headless benchmark involved running a simulator from the command line, without its user interface, and attempting to run a 5-minute long simulation as quickly as possible.

There were two types of environment:

• Small scene, where robots were put on a large 2D plane
• Large scene (screenshots below), where an industrial building model withapproximately 41,6000 vertices was imported into the simulator. Since it was not possible to import the model into ARGoS, 5,200 boxes, corresponding to 41,6000 vertices, were randomly placed in the environment

The "Large scene" environment in V-REP and Gazebo		The "Large scene" environment in ARGoS

Robot models were selected from a library of models available in each simulator, so that a sensible similarity, in terms of robot geometry and controller capabilities, was achieved. In V-REP, the original e-puck model was simplified in order to decrease its vertex count.

A V-REP screenshot of the simplified (left) and original (right) e-puck models		A Gazebo screenshot of the Pioneer 3AT robot model		An ARGoS screenshot of the eye-bot model

Three performance metrics were used:

• R: Real-time factor, calculated as R = simulated time / real time. When R = 1, the simulation speed corresponded to real time. When R < 1, the simulation ran slower than real time
• C: CPU usage. A usage over 100% indicates that multiple processor cores were used
• M: Memory (RAM) usage.

All tests were performed in a 64-bit Ubuntu Linux 16.04 environment running natively on a MacBook Pro with 4x Intel Core i7 2.2Gz, 8GB RAM and Intel HD Graphics 6000.

Because of long computation times, some experiments were not feasible to run. These cases are marked as “Not feasible” in the tables. Also notice that two values for C and M are reported for Gazebo GUI experiments, corresponding to usage of gzclient (the user interface) and gzserver (the simulator), respectively.

Simulator performance in the GUI mode, with the best (green) and the worst (red) performance indicated:

	V-REP + Bullet 2.78	V-REP + ODE	Gazebo + ODE	ARGoS + PointMass3D
1 robot + Small scene	R ≥ 1 C = 180% M = 235 MB	R ≥ 1 C = 190% M = 225 MB	R ≥ 1 C = 104+9% M = 225+58 MB	R ≥ 1 C = 7% M = 85 MB
5 robots + Small scene	R = 0.52 C = 395% M = 380 MB	R = 0.37 C = 395% M = 360 MB	R ≥ 1 C = 105+19% M = 305+58 MB	R ≥ 1 C = 10% M = 88 MB
10 robots + Small scene	R = 0.11 C = 400% M = 536 MB	R =0.099 C = 400% M = 530 MB	R ≥ 1 C = 105+30% M = 402+58 MB	R ≥ 1 C = 13% M = 89 MB
50 robots + Small scene	Not feasible	Not feasible	R = 0.87 C = 109+105% M = 1410+358 MB	R = 0.9 C = 103% M = 93 MB
1 robot + Large scene	R = 0.96 C = 205% M = 235 MB	R = 0.53 C = 200% M = 225 MB	R ≥ 1 C = 104+10% M = 264+58 MB	R ≥ 1 C = 32% M = 90 MB
5 robots + Large scene	R = 0.18 C = 400% M = 325 MB	R = 0.1 C = 400% M = 310 MB	R ≥ 1 C = 105+25% M = 333+58 MB	R ≥ 1 C = 60% M = 97 MB
10 robots + Large scene	R = 0.052 C = 400% M = 433 MB	R = 0.036 C = 400% M = 460 MB	R ≥ 1 C = 105+40% M = 425+58 MB	R = 0.86 C = 120% M = 97 MB
50 robots + Large scene	Not feasible	Not feasible	R = 0.57 C = 109+105% M = 1450+426 MB	R = 0.052 C = 107% M = 106 MB

Simulator performance in the Headless mode, with the best (green) and the worst (red) performance indicated:

	V-REP + Bullet 2.78	V-REP + ODE	Gazebo + ODE	ARGoS + PointMass3D
1 robot + Small scene	R = 4.1 C = 200% M = 165 MB	R = 3.12 C = 200% M = 160 MB	R = 42.85 C = 103% M = 107 MB	R = 300 C = 6.3% M = 18 MB
5 robots + Small scene	R = 0.38 C = 400% M = 320 MB	R = 0.32 C = 400% M = 320 MB	R = 10 C = 103% M = 130 MB	R = 150 C = 100% M = 20 MB
10 robots + Small scene	R = 0.09 C = 400% M = 470 MB	R = 0.08 C = 400% M = 480 MB	R = 5.26 C = 103% M = 150 MB	R = 21.42 C = 144% M = 20 MB
50 robots + Small scene	Not feasible	Not feasible	R = 1.06 C = 103% M = 356 MB	R = 0.52 C = 103% M = 25 MB
1 robot + Large scene	R = 1.91 C = 200% M = 165 MB	R = 0.58 C = 200% M = 160 MB	R = 18.75 C = 103% M = 174 MB	R = 15.78 C = 139% M = 31 MB
5 robots + Large scene	R = 0.2 C = 400% M = 270 MB	R = 0.11 C = 400% M = 250 MB	R = 5.88 C = 103% M = 192 MB	R = 5.45 C = 157% M = 45 MB
10 robots + Large scene	Not feasible	Not feasible	R = 3.09 C = 103% M = 211 MB	R = 1.59 C = 130% M = 47 MB
50 robots + Large scene	Not feasible	Not feasible	R = 0.60 C = 103% M = 423 MB	R = 0.03 C = 105% M = 55 MB

The verdict

Note 1: ARGoS does support multi-threading, however, it had problems with spawning threads in experiments with a large number of robots, making the CPU usage increasingly smaller with an increasing number of robots. This problem occurred in both MacOS and Linux environments.

Edits on 18/Jan/2018:
Thanks to Giovanni Beltrame and Carlo Pinciroli for alerting me to some features of ARGoS, such as the ability to interact with robots, create custom UI widgets and using Kilobots. The text has been changed to reflect this. Also, thanks to Giovanni for pointing out that multi-threading in ARGoS should work. Since it worked well for swarms of up to 50 robots, the text has been changed to reflect this. Also, the performance of ARGoS was re-tested with multi-threading on and the numbers in the results tables have been changed.

Edits on 16/Jul/2018:
Information about the conference publication has been added and some text has been changed to match the final version of the paper.