We have a wonderful old solaris legacy system that runs our payroll, hr, accounting systems. Our primary, unix savvy DBA leaves on vacation and leaves some minimal instructions for the junior, non-unix savvy DBA on how to clean out the print queue when a user submits a very large job by accident (which happens way to often). Yesterday jDBA comes running over and says I cannot ssh into the unix system — sure enough, I try to ssh in and immediately am kicked right off. Soon the helpdesk is swamped with reports of users who cannot access the system any more. I hop onto my trusty console switch and log in, only to get errors where it cannot find the /bin directory. I then start asking jDBA just what the heck he did (can anyone guess…). Continue reading