There is a running joke that doctors make terrible patients, mechanics drive the worst cars, and contractors live for years with unfinished work in their own houses.
IT companies have their own version of this.
We spend our days providing managed IT support and telling clients to replace aging equipment, verify backups, test restores, monitor UPS batteries, document systems, and avoid letting temporary infrastructure quietly become permanent. Then client work gets busy, internal maintenance gets pushed down the list, and one day you discover that the little development server nobody considered critical has somehow become far more important than anyone intended.
That happened to us.
The machine in question was never designed to be a core business system. It began life as a development environment and carried some PandaRose.com workloads because it was available, convenient, and already doing the job. Over time, more useful things accumulated around it. Nothing happened all at once. There was no meeting where someone declared, “Excellent, let us make this development box production and then forget to update the recovery plan.”
It simply kept working.
That is how a surprising amount of infrastructure becomes important.
Temporary systems have a habit of becoming permanent
The dangerous phrase in technology is often not “we have no idea what we are doing.” That problem is usually obvious enough to get attention.
The dangerous phrase is “this is fine for now.”
A development server hosts one useful thing. Then another. Someone points a workflow at it because the proper environment is not ready yet. A test database becomes convenient. A website component stays there longer than planned. The machine continues behaving, which makes every delay feel harmless.
Eventually, the server still has “development” in its mental label while carrying work people now depend on.
This pattern is not unique to servers. A spreadsheet starts as a quick calculation and gradually becomes the operational database. A shared mailbox becomes the customer-management system. A temporary server becomes the permanent home for a website. We wrote about the same pattern in When Your Business Becomes a Distributed Spreadsheet Problem: yesterday’s practical workaround has a habit of becoming today’s business-critical infrastructure.
That was our real mistake. We had allowed the technical role of the machine to change without fully changing the operational discipline around it. The environment had acquired production consequences while still being treated like something temporary.
The machine was not business-critical in the sense that its failure would stop client operations, payroll, billing, or the company itself. No client systems were affected. It was, however, hosting part of PandaRose.com, and a website is a working business system even when everyone still thinks of the server underneath it as a development box.
The data mattered. The site mattered. Recovery was suddenly worth far more time and attention than the original architecture implied.
There is an important difference between “critical” and “important enough to ruin your week.”
We had found it.
Then the power went out
The immediate event was straightforward. There was a power outage. The UPS did what a UPS is supposed to do and kept the machine alive for a while. Eventually, the battery ran out.
Afterward, the drive failed.
Hard drives have a special way of making one small noise that immediately turns an experienced IT professional into someone bargaining with God.
We investigated enough to establish that this had moved beyond normal software recovery. Continuing to power the drive on and trying increasingly desperate tools could have made a recoverable problem worse. Once physical media is failing, every additional attempt carries some risk.
So we stopped touching it.
That decision matters. Technical confidence does not mean insisting on doing every specialized job yourself. Physical media recovery is its own field, with specialized equipment, facilities, and experience. We sent the drive to a data recovery specialist we trusted and paid them to recover it properly.
Good technical consulting includes knowing which problems are within your expertise and which require different tools or specialists. Calling the right person is not admitting defeat. Sometimes it is the most competent decision available.
There is a time to troubleshoot.
There is also a time to stop powering up the failed drive and making the specialist’s eventual job harder.
Knowing the difference is worth money.
We recovered from an older backup while we waited
While the specialist worked on the failed drive, we restored PandaRose.com from an older backup.
It was not the recovery point we wanted. It was not as current or as clean as the backup chain should have been. It did, however, give us a functioning foundation while the deeper recovery continued.
That bought us time.
Once the recovered data came back, we compared it against the restored environment, recovered the newer material we needed, and rebuilt the remaining pieces. In the end, we lost approximately 200 MB of data. None of it was critical, and the missing material could be recreated or recovered through the other systems we already had in place.
The process was more manual, more expensive, and considerably more irritating than a clean restore would have been. The fact that it ended well does not make the original weakness acceptable.
It means the mistake was survivable.
There is a difference.
Git kept this from becoming a much worse article
One reason the outcome was good was that the important code and project history did not live only on that drive.
We use Git. We maintain other copies and replication methods. The failed disk contained useful data, but it was not the only place where everything of value existed.
That is what layered protection is supposed to accomplish.
Code belongs in source control. Important data should have more than one copy. Rebuild instructions should exist somewhere other than the memory of the person who configured the server. A single drive should not also contain the only surviving history of how the system works.
Git did exactly what it was supposed to do. It preserved the code and made reconstruction practical. The older backup gave us a baseline. The recovered drive supplied newer material. Other replicated copies and records helped us fill in the remaining gaps.
A worse version of this story would have involved irreplaceable code living on one machine, no usable backup, no documentation, and several people staring at a dead drive while slowly remembering that Gary retired in 2021.
We avoided that version because several layers worked even though one layer was weaker than it should have been.
That is how resilience often looks in practice. No single control is perfect. The system survives because source control, backups, replication, documentation, technical experience, and outside specialists cover different types of failure.
The incident exposed a gap. It did not expose an absence of planning.
We corrected the gap.
A backup is only a recovery plan after you prove it
We had backups. What we did not have was the clean, current, confidently restorable backup chain we would have insisted on for a client production system.
That distinction is painfully important.
A successful backup notification proves that a job ran and wrote something somewhere. It does not prove that the data is complete, current, readable, or sufficient to rebuild the system. A backup becomes trustworthy when it has been inspected, monitored, and restored successfully.
Until then, it is a theory with a timestamp.
This is one reason backup failures can survive for so long. They are often quiet. The live system keeps running, nobody feels pain today, and the problem gets deferred. Meanwhile, the distance between the current environment and the last clean recovery point quietly grows.
Then the drive dies, ransomware lands, someone deletes the wrong directory, or a power event exposes the entire arrangement.
The recovery test happens whether you scheduled it or not.
We had effectively allowed the universe to choose the date.
Very collaborative of it.
A proper backup and disaster-recovery strategy asks more than whether a backup job exists. Are failures being noticed? Are multiple copies available? Is at least one copy isolated from the primary system? Has anyone successfully restored it? Is the recovery time appropriate for what the system now does?
The cheerful green checkmark is comforting. It is not a restore test.
The deeper failure was classification
The hardware failure was the event. The deeper problem was that the machine’s classification had drifted.
We still thought of it as a development environment, even though it was carrying work with production value. Its monitoring, backup expectations, ownership, and replacement priority no longer matched what it had become.
This is a common failure pattern in growing businesses.
A spreadsheet begins as a quick calculation and becomes the operational database. A shared mailbox becomes the CRM. A temporary server becomes the permanent home for a public website. One employee’s personal workflow becomes the real company process, except none of it is documented. A small workaround keeps being extended until the business depends on it.
The technology rarely announces the transition.
There is no alert that says, “Congratulations, this temporary convenience is now infrastructure.”
The responsibility sits with us to notice.
Systems need to be evaluated according to what they currently do, not what they were originally named or intended to do. If people depend on the system, if the data matters, if downtime creates meaningful work, or if recovery would require specialized intervention, then its operational treatment needs to reflect that reality.
Calling something “development” does not make the consequences developmental.
That was the real lesson.
We fixed the system, not just the drive
After recovery, we made sure the same failure mode would not get another easy turn.
That does not mean we have defeated entropy. Hardware will fail again. Power will go out. Software will break. Humans will make assumptions. UPS batteries will eventually remember that they are chemical objects with finite lives.
The goal is for the next failure to be boring.
A failed drive should mean replacing hardware and restoring from a known-good copy. A server should be rebuildable. Backups should be current, monitored, and testable. Code should remain in source control. Important data should exist in multiple places. Alerts should reach someone while a problem is still small. The responsibility for an internal system should not disappear simply because client work is louder.
We reviewed where our data lived, how it was replicated, which systems had quietly become important, and whether their recovery procedures matched their real roles. We also addressed the hosting architecture that had allowed a development environment to acquire production responsibility without receiving production-grade treatment.
That same discipline is part of properly managed website hosting and server administration. Monitoring, maintenance, security, backups, and recovery expectations need to match what the system is actually doing today rather than what somebody named it when it was first created.
We did not write “remember to do better next time” on a sticky note and call it governance.
Memory is not a control. Good intentions are not redundancy. A checklist nobody runs has surprisingly poor disaster-recovery performance.
We changed the process.
Good IT people still need good IT processes
There is a temptation to treat this kind of story as embarrassing because an IT company should know better.
We did know better.
That is part of the lesson.
Knowledge does not automatically create priority. Client emergencies still arrive. Deadlines still move. Internal work still gets postponed because nobody is billing for it and the machine is still running. Technical skill can even make the situation easier to tolerate because capable people are good at working around small problems.
Until the workaround stops working.
Doctors know they should look after themselves. Contractors know their own unfinished project needs attention. IT professionals know backups need testing. The challenge is turning that knowledge into a process that continues to function when everyone is busy.
Self-care in a business is not indulgence. It is maintenance, documentation, training, replacement planning, verified backups, and time deliberately reserved for the systems that support everyone else.
Putting yourself last can feel responsible for a while. Eventually, there is nothing left between “last” and “failed.”
Our Build. Support. Market. model applies internally too. You cannot support clients well over the long term while allowing the systems beneath your own business to drift indefinitely. Internal infrastructure may not produce an obvious invoiceable result this afternoon, but neglect still sends an invoice eventually.
It just prefers surprise billing.
The lesson for every business
Your version of this machine may not be a web server.
It may be the accounting computer nobody wants to replace. It may be the spreadsheet only one employee understands. It may be a backup drive sitting beside the server it is supposed to protect. It may be an old application everyone still calls temporary. It may be a cloud account tied to a former employee. It may be the person who remembers every important step because nobody has written the process down.
The thing you keep postponing because it still works is often the thing most likely to surprise you when it stops.
Ask which systems have become more important than their original design. Ask which temporary tools now carry real business responsibility. Ask whether current backups have actually been restored. Ask whether code, documents, and operational knowledge exist somewhere other than one machine or one person’s head.
Most importantly, ask whether the recovery plan reflects the system you have today rather than the system you intended to have three years ago.
We tell clients to verify backups, document critical systems, replace aging equipment, monitor infrastructure, and plan for recovery. We learned, somewhat loudly, that those rules still apply when the client is us.
If you are unsure whether your backups are clean, your recovery process is current, or one of your temporary systems has quietly become production, talk to Panda Rose. We can review the environment while the problem is still ordinary maintenance rather than the central plot of your week.
We made sure this exact failure will never happen again.
Until, of course, next time.
At least next time should be boring.
