Automation – When Comfort Turns To A Burden
When you do it once: do it.
When you do it multiple times: automate it.
Similar comments are choking the sysadmin threads.
Automation is a standard.
- Deployment.
- Configuration management.
- Source control – IaC, key stores, passwords.
The more we rely on automation, the easier it becomes to overlook a hidden cost.
The abstraction hides the operational complexity.
The tool that makes infrastructure easier to operate can also make it easier to stop understanding.
The operator gradually loses touch with the infrastructure beneath the abstraction.
You work with multiple servers. Some of them are similar, some are unique.
Manual management of the fleet is already impossible.
You attended at a conference where a good deployment and automation tool was presented.
You remember the flowing configuration of the projected video.
You were amused by the simplicity. The abstraction that orchestrated a whole datacenter.
You introduce the tool in your environment.
The new automation code repo is growing.
Team mates come and go, they add their own parts.
The abstraction itself became complex too:
- conditional control structures
- loops
- template files
- conditional includes
People are hired and fired – you remain the constant: you still manage the environment.
Fast forward five years.
The Problem Surfaces
Your configuration management works. You hardly touched the system by hand.
Everything works as expected.
Then one night you get a call: the retail part of the environment is unreachable.
Even worse: the automation services are down as well.
You try your best to stop the losses and revive the services.
Finally you use SSH to log in and assess the damages.
You hardly remember the exact paths of configuration files.
You struggle to issue basic commands.
What happened?
Your operational knowledge eroded while the complexity was hidden.
It’s not the fault of automation.
It’s not even the fault of the environment.
It’s a missing step in the sysadmin procedure – to retain operational knowledge.
- Time passed by.
- Tools were updated – new configuration was introduced.
- The automation silently hid this from you.
- The system became outdated – while it was functioning well.
- You did not touch the console – your knowledge declined.
Stop here for a moment: you can argue it. You can deny it. If you're an operator - you were there. You experienced it to an extent.
The Possible Solutions
Obviously the solution is not abandoning the automation.
The problem is not the automation itself.
It’s the missing hands-on experience.
Your operational knowledge must be retained with practice.
Introduce processes to examine:
- software upgrades
- configuration references
- changes
- new commands
Automation is good.
It standardizes the deployment and configuration management.
Infrastructure as Code can improve operational stability.
Automation without a knowledgeable operator is like a katana in the kitchen.
Someone will cut themselves.
Final Thoughts
Cut the corners. Automate. Script.
But always retain your knowledge.
It’s never a bad thing to refresh the memory.
Time spent practicing in a lab is never wasted.
Sometimes we can forget even the basics.
Document. Practice. Improve.