Some days ago I started roscore and got faced with an error message like that:

process[rosout-1]: started with pid [16089]
[rosout-1] process has died [pid 16089, exit code -11, cmd /opt/ros/indigo/lib/rosout/rosout __name:=rosout __log:=/home/rosuser/.ros/log/9b3b3980-0b60-11e4-80f9-0015afdb2ab9/rosout-1.log].
log file: /home/rosuser/.ros/log/9b3b3980-0b60-11e4-80f9-0015afdb2ab9/rosout-1*.log

Okay, so roscore seemed to have crashed and created a log file according to the given path. But the logfile was empty *please imagine dramatic sound effects here*. But what do you do, if a programm (rosout?) crashes without log file and an error message like the above?

You insert your well done system wide backups like snapshots from zfs,btrfs, virtual machines, lvm or anything. If you forgot to do so… *fail sound here* you might need to manually check what changed since the last time it worked.

And so I did. For several days.

I soon figured out, that this error applied to all rosccp related ROS-programms – but left all rospy parts alive, which finally put my on track that there has been an kernel update of my ubuntu 14.04, which I unfortunately installed in a moment of weak decisions.
So reverting that would have been been an option, further my lib-boost version seemed to have changed – since ROS is very sensible to that, this might have been the problem. Therefore I tried everything by manually reverting updates, reinstalling packages, recompile everything from source, searching system logs and soon really considered to reset my system by installing good old 13.04 with ROS hydro…

But wait! Sometimes you strike lucky and time solves all issues. Today I’ve just upgraded my ROS from their repos and tadaaa – everything works again.

But why do I write this into an post? Because its easy to avoid situations like that and I want to share my hard learned lessons with you the easy way:

1. Get your ROS version straight – Do you really need the latest ROS on the latest kernel?
   The answer for your system is probably no. I am currently running stable ROS Hydro and ‘unstable’ ROS Indigo on Ubuntu 14.04 on latest kernel. It works – but it would have been way easier to stick on Hydro all the way.
2. avoid apt-get dist-upgrade on critical ROS machines
3. use backups and / or virtual machines
4. rospy didn’t cause any problems so far – in case performance isn’t the most important thing, think about using python
5. avoid to put all your catkin_ws code into one git repository  if its running on multiple architectures (x86,x64, arm6,arm7) – alone the openni2_driver took nearly all my sanity during learning that lesson….

That is enough for today, but after this list, I really think about tracking all hard learned lessons in more public and better organised location – ROS best practices? We’ll see.