A server that wasn't meant to exist

(it-notes.dragas.net)

184 points | by jaypatelani 5 hours ago

16 comments

  • johnklos 4 hours ago
    I've seen situations like these before. This is why off-site backups are so very important. I've also been in the same position of providing data from a backup that someone was intentionally trying to destroy to escape responsibility.

    This story even hints at a common theme that happens even when people aren't trying to destroy data - that some people will tear down whatever they inherit, then blame their predecessors for the problems that result.

    • apples_oranges 2 hours ago
      But if you don’t blame them it can also backfire. I inherited a bad codebase once and tried my best to improve it. But there was only so much time. When I left the guy after me blamed me for the still bad parts immediately.
  • duxup 4 hours ago
    It's always interesting to me how easily corruption occurs. I always assume that accounting double checks things and so on, but I've seen so many business where someone just creates an account and money goes out and ... nobody notices for years.

    I've even created automated invoices for some companies and realized that some data was missing for months. And yet they got paid significant amounts. I realized that the invoices could have been for just about anything and they would have gotten paid ...

    • forinti 3 hours ago
      When Robert McNamara took over Ford, accounting was so messed up, they would weigh their invoices and if the amount wasn't too far off from the expected dollars/pound ratio, they would pay it.
      • duxup 2 hours ago
        Even Google evnetually caught a few people who just cold sent in invoices and found that Google would pay.
      • ngangaga 1 hour ago
        I had no idea that piece of shit was associated with Ford at all.
        • forinti 1 hour ago
          He talks about it in The Fog of War (which is not going to make you change your ideia about him).
  • jagged-chisel 5 hours ago
    I think I missed something. They later offered the guy the world to solve problems. He declined and then complains they wouldn’t provide the tools he needed.

    Part of “name your price” should include whatever tools - up to and including ownership of processes.

    • kentm 5 hours ago
      Yeah I think something was missed. My wild speculation is that the person thats "causing issues" has a privileged position with the owners. The owners are unwilling to completely cut this person out of the business, and that is what he means when he says that the owners won't provide the tools he needed.
      • hengheng 4 hours ago
        My mind immediately went to organized crime. Money laundering for people who he rather didn't know his name.
        • draga79 4 hours ago
          I’m the author of the post. I hinted, in a cryptic sentence near the end, that I necessarily had to leave out the worst parts of the story. No, no organized crime. But yes, there were people who appropriated resources that weren’t theirs and used every tool at their disposal to avoid scrutiny. To keep it vague, let’s just say some of the people involved had means that could seriously harm the businesses and their owners. And since these were primary businesses, that would have been a serious problem. The owners, knowing this, tried to find solutions but couldn’t really “afford” to remove the people involved. To be specific, in the end the owners themselves were aware of what was happening, but hoped to resolve it with a few more checks. Eventually, I realized that as long as there was enough money for everyone, they were okay with the ongoing theft.
          • Aloha 3 hours ago
            Its better to know who is stealing from you (and how much) than not - sometimes the evil you know is better than the evil you dont.
          • sidewndr46 1 hour ago
            if you weren't getting paid for this - why get involved in the first place?
            • draga79 1 hour ago
              I've been paid for doing my job: creating the infrastructure, configuring stuff, etc.
              • sidewndr46 56 minutes ago
                Alright, I was thinking this was just the case of helping a friend
        • lisper 4 hours ago
          This looks like a clue:

          "I even worked on translating Archivista’s interface into Italian, since it wasn’t yet localized, just to make it easier for users."

          • draga79 4 hours ago
            No, that's not a clue :-) I've just replied, clarifying this part, to the previous comment
            • bombcar 2 hours ago
              Got it, it was disorganized crime, not organized. ;)
              • draga79 1 hour ago
                Yes, exactly :-D
    • poyu 2 hours ago
      I thought that just means they're buying him out?
  • BubbleRings 3 hours ago
    Great read! Yeah, these days if I get asked for technical advice, I’m always glad to put good effort into suggestions. But as soon as you tell me “well I want to follow some of your advice, but I want to do this other stuff the wrong way”, I usually say “Good luck with all that!” and away I go.
  • bzmrgonz 2 hours ago
    How you gonna leave out the good parts like circa<year> so we can gauge the tech available then? Also, what about the tools you used to sync/backup to owner's house? My personal query, why did you move to freebsd? was it a different application/use? This is an awesome story, our modern approach would be to install nextcloud/owncloud with collaboration and rsync/syncthing to an offsite NAS (owner's house). As for your decision, I would have agreed to a directorship and hired a local MSP to do things the way I wanted. This would have allowed you to have your cake and eat it too. A lot of times, in these situations, all you need is trusted eyes and ears from outside the corrupted fold. This principle is used in the military and diplomatic core, there is a staffing structure, and then there is an XO, who is hired and controlled from HQ. This XO answers to HQ, not the local structure.
    • draga79 41 minutes ago
      We're talking around 2009 — I don't recall the exact period, but that’s the era. For backups, I used rsync-based syncs and kept history by using hard links and rsync on top of those. I also had a Perl script that automated the whole thing, but I’ve long since forgotten its name.

      As for the rest — I hear you, and I totally agree. But at the time, I was young and more focused on building things with healthy clients who genuinely wanted to create something good, rather than trying to salvage a situation that, honestly, was nearly beyond saving.

      I switched the ALIX to FreeBSD for other tasks, and FreeBSD (with its native read only support) was perfect for the new workload.

  • clysm 2 hours ago
    Why the hell is there a line break after every sentence?
    • amatecha 2 hours ago
      Yeah, that's really a strange choice for formatting and makes it very hard to read. Not the typical practice to insert a <br> after every sentence... (that said, the post itself is a great read!)
      • draga79 58 minutes ago
        The goal of truncating the sentences in that way was precisely to increase the suspense a bit, but I believe I miserably failed, making it just less readable.
      • frizlab 1 hour ago
        I think it’s called ventilated prose. More commonly found in code comments.
    • robohoe 2 hours ago
      Hello my high school research paper teacher
  • draga79 3 hours ago
    Author's note: Many readers, understandably struck by the severity of the events, have speculated about the involvement of organized crime. I want to clarify that, while the situation was extremely problematic and dishonest, that wasn't the case. The "worst parts" I alluded to referred to other internal dynamics, abuses of trust, and improprieties that I prefer not to detail further for privacy reasons and to avoid weighing down the narrative.
    • Boogie_Man 2 hours ago
      Dear Abby,

      My dear husband of 40 years has recently passed away, and while I'm still grieving the loss, his former gay lover has been syphoning money from our company and threatening to expose my husband's bisexuality (NOT socially accepted in Italy) if I expose the theft! I've hired an IT kid to prevent further theft, but it turns out he has some sort of connection to the thief as well! I'm totally at a loss as to what to do.

      -Benedetta Bothways

  • forinti 4 hours ago
    Reading through it I had a feeling it was in Italy. I was bit sad to confirm it.
    • draga79 4 hours ago
      Italy, but no organized crime involved
  • freehorse 3 hours ago
    Probably there was a lot the family did not know about the deceased father.
  • dgfitz 3 hours ago
    Can you elaborate at all as to why you didn't make the phone call you eluded to that made the other person change their tone? I assume out of respect for the deceased/leaving skeletons in the closet?
    • draga79 1 hour ago
      Sure, I can say this. The person I would have called, someone very close to me, would have been extremely disappointed to learn what was happening. They were very proud of having helped, during difficult times, the very person who was now threatening me. And since this person close to me was facing serious health issues (though still had authority), I chose to avoid causing them further pain that, ultimately, would have been pointless at that moment.
  • NKosmatos 3 hours ago
    > Because sometimes, dishonest people do win.

    Let me fix this for you… Because always, dishonest people do win.

    Good read and it would make a good short film :-)

    • mulmen 2 hours ago
      This is needlessly negative. It’s clear that dishonest people do not always win. Disproving such a claim requires finding only one case of a successful prosecution for fraud.
  • asmodeuslucifer 4 hours ago
    good read.
  • 01HNNWZ0MV43FF 5 hours ago
    Creepy
  • immibis 2 hours ago
    > Because sometimes, dishonest people do win.

    Dishonest people almost always win.

    Not any individual one - a particular dishonest person might only win 20% of the time - but in aggregate - the winner is almost always a dishonest person.

    Even when a game rewards honesty, dishonest people are willing to be honest if that's truly what gives them the greatest chance of winning, so they still win.

    • mulmen 2 hours ago
      Is this some kind of inverse no true scotsman?

      If you win by being honest that’s not dishonest.

      • aoki 1 hour ago
        I believe they are saying that there are multiple rounds, each with different games - some with honest optimal strategies and some with dishonest optimal strategies. A dishonest person can always choose the optimal strategy for each game, but the honest person can only choose the best honest strategy. So in aggregate the dishonest person comes out ahead.
        • mulmen 1 hour ago
          Ok but people are both honest and dishonest so how do you decide what type a person is?
  • renewiltord 4 hours ago
    Fantastic war story. There's always like these dozen hangers-on who've made their fortune parasitizing successful people.
  • vaadu 3 hours ago
    [flagged]
    • brazzy 3 hours ago
      DOGE is doing no such thing. They are destroying things they don't understand, with a political agenda that has nothing to do with saving money, and doing immeasurable harm to the USA in the process. The link contains nothing but bullshit and lies. It's also guaranteed that many of them are using this opportunity of unfettered, in transparent access to illegally enrich themselves.