Google locked our company out of using our own domain on Google Workspaces because a former employee had signed up for a Workspace account using our domain, and we had no way to recover it. We literally manage our domain in GCP but they wouldn't accept that we own our own domain and thus won't let us use our domain with Workspaces, which screws you out of a lot of Google functionality. Never seen a company that was too incompetent to allow users to use its own product and make zero attempt to solve the customer's problem.
Google Cloud CEO Thomas Kurian said "We need scale to be profitable." No, you need to stop treating your customers like crap to be profitable. We don't trust you, you are a pain in our asses, and we have alternatives, so there is really no need for us to spend our money on you. It's now a matter of when we all move to AWS and Azure, not if.
Sorry to hear that happened to you. They do lock out domains that were previously registered and still active from being re-registered [that's happened to me too]. Their Support should have offered you to contest and take over. It's not immediate [takes several days to go through the process and the departed employee will delay things] but if you can demonstrate positive DNS record control and go through the motions with support it does get done and you gain control. If they didn't offer it to you over support, try asking for it [if you are still wanting to go ahead with it]. It's not great speed but it also serves to protect you should someone else at the company decides to impersonate you and recreate the domain.
> if you can demonstrate positive DNS record control and go through the motions with support it does get done and you gain control.
nope. made the DNS change. went through the motions. even got at call - at 5AM Eastern Time - from a support person to ask me one question, then "I need to confer with so and so". several emails later, new support people asking the same questions. but because I cannot answer "recovery questions" made by person who has left the company, they refuse to offer any other support. attestations of proof of who we are be damned, they just follow their robotic script. so we're just screwed.
> attestations of proof of who we are be damned, they just follow their robotic script. so we're just screwed.
Hire a lawyer and send a certified letter with return receipt requiring them to explain their decision, and prepare to sue. I'm presuming that this is a B2B dispute, and I notice that a lot of HN people are (rather counterintuitively, considering Y Combinator owns HN) scared of hiring lawyers. Unless that the records show that your old employee is the recorded administrative owner of the domains (this unfortunately happens frequently to small businesses), this should resolve this issue.
That’s completely contrary to my experience. Running a small startup we encountered all 3. For us, MS/Azure were the most eager to please but in many ways the worst from an execution standpoint. GCP were very good from an execution standpoint but absolutely zero support and stories like this one scared us off. Lots of big GCP corporate clients actually use them through consultancies which essentially provide the support and are large enough on aggregate to get listened to by Google. AWS were good at actually doing things and also offered support the couple of times we needed it.
This has been my experience too: Google Apps/GCP might have marginally or even substantially better products than Azure/365 but Microsoft almost never kills products and force you to migrate to their newer offering and they have excellent support folks
Microsoft has learned through the decades where you can make big bucks: charge enterprises through the roof for “basic” stuff, and in return promise them to _not_break_their_shit_ and help them (for even more money) if they screw up themselves.
Now with Azure, basically they’re making that available to smaller scale players as well.
Meanwhile Google operate GCP in fundamentally the same way they operate free Gmail accounts: hey, this stuff is super neat! So neat actually you bunch of freeloaders are not allowed to complain if this free and neat thing spectacularly breaks once in a while! Wait…you pay for this stuff? Naaah!
I had an incident a couple of years back where AWS couldn’t swap a cancelled CC on AWS after we had for an unknown exception, and we kept getting overdue invoice alerts. and it was during December holiday season. AWS singapore has to escalate to US since it was a billing issue.
A false negative security alert from 6 months earlier was the cause, which had blocked CC changes. It took months to resolve but it was manageable.
I know from first hand from a different experience that you can be months overdue with tens of thousands of missing payments on AWS, and tens of thousands in recurring costs without them closing your account. You can effectively run a quarter of credit line from your hosting costs without any issue if you need that.
Your account manager doesn’t even start raising it personally for the first 3 months, they are very easy to deal with.
You are experiencing a false equivalence here. You are attributing outcomes (as being the same) as being the result of some common factor, when the cause is a very different factor.
Google is big. As are Microsoft and Amazon. Google has bad customer service, therefore so do Microsoft and Amazon.
You tried to x in the cloud. You had a bad experience. Therefore the cloud is bad.
In reality of course big companies are not the same. Bigness does not imply sameness. Google support is well known to be rubbish. Microsoft and Amazon support are well known to be excellent. If support matters to you (like when things go wrong) then take that into consideration.
Same for the cloud. There are advantages and disadvantages sure, but providers are different. Google likes to kill services. Others don't. Startups vanish as often as they appear.
Tech is not the only place where you see this false equivalence. All politicians are the same. All fast-food is equally crap. All cars have the same gas mileage. Turns out, no, they are very different, but you need to dig deeper to understand their differences.
Wow I'm really sorry this happened to you. I do believe your conclusions are correct about AWS. Azure generally seems fine too.
Google's aversion to customer service makes them extremely dangerous as a cloud provider in my mind. This also goes for other critical business services, like Gsuite (I know! It's convenient).
If you have GCP or Gsuite now and you're trying to evaluate how big of a deal it is my suggestion would be to pick up a phone and attempt to talk with someone at Google about your account. This experience can act as a preview of what that process might look like when services are turned off.
If you try to call Amazon on the other hand it feels like Jeff Bezos might hop on the call if things aren't going well.
I've had some bumpy points with Azure, but if I really can't fix something I can pick up the phone and they'll try to help. The idea of having critical business services with a company that you can't actually talk to is terrifying.
Jeff is famous for sending “?” Emails to his s-team of direct reports for customer issues they mailed him about. Those ? emails derailed many a roadmap.
I know it’s just anecdotal, but once you have some spend (around 1k a month I’m guessing), you get an account executive who I’ve been able to reach on the phone whenever I’ve needed. In fact they tried talking to me on the phone more than I wanted.
My rep was able to get me connected to a few tech leads on various cloud products and one time when we were hitting weird quota limits the team protectively reached out to me to ask what the heck we were doing. Maybe we were just lucky, but I’ve had similar experiences in AWS and Azure too.
One time I emailed jeff@ and then a few days later someone called to talk to me about my Kindle on Android bug. I got a workaround and it was fixed in the next version.
And I've talked to a support human at Google before, about our GSuite.
I’ve had phenomenally good support from AWS. I recently had a problem where a nat instance was opening more than 55k simultaneous connections (don’t ask) and AWS support helped me track down which ecs service it was coming from. I’ve had utter shit support from GCP (mostly around “hey can I get this API limit raised). My best support from GCP was from our customer evangelist. He actually knew stuff. His turnaround time was 3-5 days. I eventually just stopped filing support cases because they were useless.
Also currently actual AWS staff is answering on the new community support forum re:post, I’ve been taking full advantage of that.
There are some things which AWS suck at unless you have an enterprise support plan, like having a deep review of a technical issue in their products, and having visibility into product bugs in general.
Even then after reporting a significant issue like Aurora query execution being non-deterministic in a very specific case, I only found out it was solved months later after working around it from reading patch notes.
In the beginning of the year, I was recommended by a friend working at Google to test GCP using the tutorials on https://www.cloudskillsboost.google/
Google banned my account while I was doing one tutorial. One step failed, the provisioning of something, and I had to redo a few things again. It succeeded but I had used more egress bandwidth than allowed to run the tutorial correctly so they banned me automatically a few minutes later.
The support unbanned me after some time but I thought it was a sign to not use GCP yet.
This is really sad. I feel bad for you and hope you manage to rebuild
your base somewhere else safe - hopefully with technology that you
control.
You won't be the last person to get screwed over by the callous
machinery of dehumanised big-tech (AWS or Microsoft would be no
better, sadly I think you are rationalising loss). We must all wake up
to just how horribly precarious the lives of ordinary people are
against the arbitrary whim of opaque, unassailable power centres that
masquerade as "Free services".
Aws was/is extraordinarily careful when it comes to suspending accounts, even for non payment. Only verified malicious use of their services resulted in suspension. There are probably edge cases I didn’t get exposed to, but stuff like this didn’t happen and if it did I’m 95% certain they would take it very seriously and resolve it asap favorably to the customer. Amazon has a leadership principle around customer obsession, and it generally resulted in making things better for the customers even at Amazon’s expense (I.e., prolonged non payment not resulting in suspension and rebates and credits back for bill shock, etc). Again I’m sure folks can find counter stories - it’s an ultra mega corp for sure. But the culture was definitely oriented on our customers favor.
That wasn't my experience. They banned me while I was trying things out, for some baseless suspicion of an infraction. Just rented some cheap server from Hetzner, did the job well.
They weren't even happy with my verification using a live selfie with my passport (i.e. by live video camera stream). I think they didn't like that I ran my dedicated server for about 6 hours and then shut it down.
I was testing software for the upcoming macOS Ventura release, and I only needed 6 hours of initial usage, and then 6 further hours a few weeks later (when a new developer beta dropped) but they closed down my account on the second day.
Have the same experience with AWS, the customer service is nice and I always felt like I’m talking to a human and not a bot like with Google. Google is just the worst nightmare for developers
dvery time the Google Cloud sales people contact me to try to increase my enterprise's use of GCP, I tell them this over and over. I don't want to give my money to Google if AWS does a much better job of customer support. And we're serious: we use all 3 clouds, we have TAMs in each of them, and a direct line to senior leadership.
I used to work for Google and then left for a startup; the sales people kept calling and saying "Urs wants to talk to the CEO" and I'd say: "the CEO hired me to negotiate our cloud deal. We considered GCP but AWS made a far better offer. Urs can call me and we can discuss the limitations of GCP". (I know Urs; I'm the one on stage with him when GCE was launched, running the demo). He never did call and we never gave them our business.
(BTW, my first question for Urs was going to be "why did you defend killing Google Reader? That strongly affected my interest in buying Google products." :)
Urs should only be making technical decisions for the sake of the technology. When it comes to business, he comes off like a cartoon supervillain business man. He really doesn't listen to how his words (or tweets) land.
The TSE should have been able to override the accruing charges on their GCP account. Or they can use another payment method to pay the GCP bill. It sounds like funds they are owed, that they would use to pay GCP bill are locked away until the account is verified.
I don't have the statistics to back up any statement about the odds that one provider or another is safer, but AWS has had a person I could talk to and they really resolved my issues and seemed empowered to do so. That's a big and real difference.
Anecdotal but I listed a few graphics cards on Amazon with the same account I had AWS on and they booted me without any recourse or contact information.
There really needs to be some more legal precedent around this stuff. In the commercial real estate leasing industry there are legal commitments and a framework of legal understanding to protect business owners or at least provide predictability. That needs to be applied to leasing computing resources given how much business flows through the internet now.
The frameworks largely exist, the above situation is like if you rented an office building and then started running an unlicensed side business out of it. It’s just a lot rarer (cloud computing has a much lower cost of entry then a commercial lease).
I agree. But I can't imagine what that would look like. Arbitrary
discontinuation of service is a tangible harm. But how could one
possibly compel a provider to do otherwise?
You'd have to do something like construe the provision of service as
an implied contract to continuity in the same way that T&Cs assert
that usage of a system implies acceptance of some rules.
Customer Obsession
Leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers.
10 years across all 3 major cloud providers (and some others like Rackspace, etc), I can tell, Google support has been the absolute worst of all. I don't say that lightly. I take pride in how I can deal with customer service the most peacefully in my org. But that's all out of the window when their CS is nonexistent.
I can't remember the number of times they've shut down someone's production workload without notice or warning, gave them no way to resolve/rectify any supposed infraction, and gave no way to contact them to appeal the decision.
I once had to resort to numbering my points and just referring the boilerplate responses (few and far inbetween as they were) back to the numbers over and over until I finally gave up, moved away and never looked back.
> Google is famously indifferent to user support issues
Of course they are. They have the BEST engineers on the planet! How could they ever be wrong? The produ... the customer is obviously wrong in a way they're too dumb to understand.
Amazon and Azure have better support for everyday developers, especially around issues like this. Have used all three. It's not just a running joke--Google is culturally allergic to staffing support seriously for anything but ads.
Yes, but cloud has such a good chance of becoming a profit center. They should be developing it and staffing it accordingly now. Amazon was able to think ahead in this way. Amazon also has significant cultural weaknesses, but their drive to improve the user experience forced them to stumble into principles of good customer support. Google has this drive to test and improve the interface only.
My first experience with AWS was around 2013 when I accidentally left a (cheap) ec2 instance running for months after I was done experimenting with it. Support refunded me no questions asked. Fast forward 2 years and that experience was an important factor in my decision to run servers costing 6 figures annually with AWS over their competition.
I came to write a very similar story involving a lazy error on my part (no error checking on a shutdown ec2 instance script, resulting in a 64 Xeon, 8 Nvidia Tesla server running 24/7 for about a month) and an astronomical bill resulting.
I had a rudimentary batch job system running. Since I was charged for each power event, I wanted to minimize the power state changes and just queued up the work that needed GPU acceleration until there was enough to justify the cost of at least an entire hour and two power state events. When the cheap, small, always on instance received all the work output from the giant beast server, it would then send a power down command using the aws cli. Yeah, like I said, I was lazy so I just sent off the request and small server just went on about it’s business. Oops.
Like I told the aws billing rep, it would have bankrupted me, ending that venture. I asked that, since it is all virtualized, that while yes I had the virtual hardware provisioned for all that time, if they could get someone to look at the actual utilization they would see it was essentially none. They got back to me and in a day or so to let me know they had forgiven the entire amount.
AWS support actually listen, do real work, and very often it seems, do what is in the best interest of the customer even as it works directly against the company’s bottom line.
When I was early in my career freelancing my AWS account was shut down for no payment for over a year, more than half a decade later I re-opened it through support and they wouldn’t even let me pay my old bill.
I’ve been able to reduce reserved instances when the market has turned unpredictable.
AWS isn’t a pushover, not even a little bit. Dealing with them externally has just been a good experience with professional understanding and a flexible business sense.
Well... AWS has better support IMO (like actual people you can talk to).
But I still chose to not tie myself to any of these platforms for anything critical. It requires additional upfront effort, but I can migrate all my applications off of AWS in a moment.
There is no easy way around Google and Apple when it comes to your app staying available, but I am trying to minimise any chances I could be flagged for anything (basically not linking those accounts to any other services).
> But I still chose to not tie myself to any of these platforms for anything critical. It requires additional upfront effort, but I can migrate all my applications off of AWS in a moment.
Then you are probably not doing "cloud native" development using their proprietary SDKs, so probably just running some EC2 (virtual machine) instance and RDS (database) instance. That is about the most expensive way you can use the big-tech clouds, at that point you might as well move to the smaller hosting providers like Linode.
That is not what I would call a "moment".
If you have to rewrite your application code to migrate to a different platform, that's not going to be a moment.
Even if you abstract away most of the platform specific stuff in your code, that's going to take days/weeks of implementation and testing before you can go live. That won't help you when a provider suddenly bans your account in the middle of the night, and you need it running asap.
> If you have to rewrite your application code to migrate to a different platform, that's not going to be a moment.
Having a couple of days of downtime might be an acceptable tradeoff for an event with a very low chance of happening. Risk management basically. (If your business doesn’t survive that downtime it might be a completely different story of course.)
Yeah, people frequently overestimate how reliable their app needs to be.
My CEO is constantly beating it into my head: "I will gladly accept a day or even more of downtime from time to time if it lets me get what I want 10% faster".
It does not mean shoddy engineering. It just means consciously deciding not to do some work. Spending couple more days to get the app automatically fail over to another without downtime? Engineer it to run as a distributed system when it could happily run on a single server? Let's just not do it, let's make it simpler and easier to make and accept that if AWS has a failure it may take a moment to spin another instance somewhere else.
How well do smaller / more traditional providers support auto scaling and various other "elastic" features of AWS? I only have experience with renting a "traditional" VPS, and some of the more modern AWS offerings.
GCP is not going to be a thing in a few years. Why would anyone pick GCP at this point with all the horror stories coming out of there? Yes, support is hard, yes it's expensive, yes it takes time. But at the end of the day this is how you make or break the trust of your customers.
Sadly we hear so much of this on a daily basis as we are often sought out to support suspended Google Business Profile (GBP) accounts.
It's not uncommon to have these same small to medium sized business owners ask for help with other suspended accounts in other Google apps like GoogleAds or Google Merchant Center.
The support across all platforms is a growing eyesore for Google, and at this point I hope becomes the catalyst for real change in this space.
It was bad enough when Yelp would act like the Mafia in regards to how they approached and treated businesses, but this level of negligence (from such a tech giant) is something to really see.
We've been advocating on behalf of all these small businesses for years and truely hope that the outcry grows to a fever pitch where no more band aid solutions are used anymore.
To quote a HN user talking about another similar instance of a cloud service failing to support their customers:
> “You've got an additional problem though, which is that this tells us you have two support channels: one that doesn't work (i.e. yours, the one you built), and one that does (Twitter-shaming). The first channel represents how you act when no one's watching; the second, how you act when they are. Most people prefer to deal with people for whom those two are the same.”
not all heroes wear capes, but look at this person helping out on thanksgiving. this is both amazing and sad (that is necessary to do this to get help)
its nice on a personal level but a sign of how broken things are at GCP that an angel has to emerge from the clouds just at the last moment to save your livelihood? that is super creepy and not the way the information economy is supposed to work, in my opinion.
Same thing happened to me: “suspended, need extra verification” after YEARS of the account existing just fine. Nothing helped. Not doing what they asked (multiple photos of photo id), not emailing, nor calling support.
I ended up calling an ex-coworker who still worked at google. He sent an internal email. It was resolved the same day. I never found out what the issue was.
Don’t use google for anything you care about, unless you maintain close friendships with googlers.
That doesn't always work, (following anecdote is not about google cloud): someone I know lives in a street that's egregiously and very obviously broken on google maps (whole network of streets mapped with the same name, private pathways mapped as public streets, whole bunch of repeated door numbers, etc) and since everyone trusts google maps for everything this has caused a bunch of serious issues for the person as they interact with businesses and even government. Any attempts to get this fixed through the normal process get ignored (I have personally tried, several years ago, twice).
Over the years I asked two different friends at Google for help, one of whom was management. They were baffled when the internal message thing yielded no results either. There doesn't seem to be any process for getting certain types of issues resolved at google.
My point is that google can ruin people's lives even if the victims aren't the ones using it. Gmail blocking inbound mail is another example - if gmail decides to block your service, everyone will blame you, and there's nothing you can do about it other than wait.
It's sad none (I think?) of the big cloud providers doesn't allow for prepaid account, it would solve both "our idiots thought that blocking long-term paying customers with no human intervention is a good idea" and "I woke up this morning with $20k AWS bill" problems
Aws has no provision for shutting stuff down for exceeding a budget (or at least when I ran services there). We would rather reverse your $20k bill and keep you alive. I ate huge write downs from small and big customers accidentally blowing themselves up. I remember on guy almost in tears because he knew he would be fired due to his screw up that blew up the bill. It made me proud to help him. Amazon has a ton of serious issues but they are genuinely incentivized to care about customer problems (I.e., you are paid and promoted on how far you go for customers).
> We would rather reverse your $20k bill and keep you alive
And as a customer, I never want to be in a position where I ever see $megabill. Maybe I might be able to get reversed if the gods smile upon me that day, but it would be incredibly stressful until I had resolutions. Seems an obvious blindside for “customer obsessed”. Anything I might host is practically a joke for which I would gladly have the provider pull the plug if billing suddenly spiked above pre-established threshold. I’ll take the downtime if it helps me sleep at night.
I've had the exact polar opposite experience on any company running systems with paying customers. Usually it's "don't care what it costs, just fix it" which lines well with AWS' behaviour on this. When downtime costs tens of thousands of dollars _a minute_, people tend to stress less about some inefficiencies like this
And those customers can choose to ignore the emergency billing shutoff. The rest of us need assurances we do not have an unlimited liability that could strike. My potential spend is under $1k per year. Vastly different cost considerations at play.
One qeustion is whether cheap cloud services even make sense for an AWS, as opposed to a Dreamhost (targetting smaller businesses/individuals specifically) or Hetzner/DO.
I would imagine AWSs profit is in servicing large clients using lots of CPU/bandwidth, so the only reason to even bother with small clients is some kind of low-probability funnel for the occasional startup that blows up. Or just a loss-leader strategy for market domination (but only if that small-biz experience is vastly superior to other clouds' offering).
One way or another, small businesses should never trust that big businesses are going to look out for them, for any number of "rational" reasons on the big biz side, especially if it's not clear you're their bread-and-butter.
Everyone praises their managed kubernetes offering. I have no experience with it, but if you ask people around, everyone prefers GKE to alternatives. That might be one reason.
yes. it's a no brainer really. you get GKE and in return you may lose your business if you hit any snags. we're past the point where K8s is this exotic thing where people are not sure exactly where it is.
At this point I am having trouble understanding who would consider GCP. I mean maybe something along the lines of: something something, multi-cloud strategy, no lock-in options, let's also use GCP so we have multi-cloud. But I never see it as the option to pick. It's sad really.
1. SaaSy kids in SV or who wish they were in SV who also think Google Sheets beats Excel.
2. Big enterprise executives who heard Google was what the cool SaaSy kids do from someone in I.T. then got lured by too-good-to-be-true discounts from Google Cloud sales team as long as the executive is happy to be featured for their forward-thinking tech chops in pressers about picking GCP. To their credit, they will undo the Office to Workspace migration mandate almost immediately, after making their 140,000 users attempt it from IE 10 on Win 7, but there won't be press about that.
Sigh. It's now been [0] days since the last "My [business | e-mail | critical thing] depended entirely on a single service with no backup, I got suspended, and now I can't do my [business | e-mail | critical thing]!" story. I do sympathize, and the right fix is for these services to have better support, but come on... These stories happen so often and get ample press. They should not be surprising. Please stop doing this, people! If something you rely on for your livelihood depends on a single company as a single point of failure, that is a flaming, burning emergency that needs to be solved right now!
are you supposed to write your own firebase? develop your own android?
yes, OP learned a hard lesson about Google, but that does not mean that you cannot build stuff using just one provider. you need to do your due diligence of course. You need to have a plan, of course. But the cost of doing this on 2 platforms skyrocket. And especially for a small player the multi-cloud, massive investment is just not an option.
I'll go one step further and say that if your website/app/service can be taken offline by any issue with a single vendor, it's more than an emergency, it was engineered incorrectly from the start.
Are new devs taught that having something in two AWS zones equals redundancy? You drank the Kool-aid, people.
I’m sorry. Are you able to work with your customers to keep their business while you sort this?
I would port your stuff to aws. Most services have close equivalents and it is probably worth while to try. As another poster noted it might be useful to have cross provider redundancy.
I’ll be honest I have refused to work at gcp and have steered the mega corps I’ve advised on cloud choices explicitly away from gcp. Azure is better, but less technically sophisticated. It’s still a one shop cloud. And any company whose core business is surveillance and advertising won’t care about their customers - you’re just meat for their machine. Amazon was built around a customer centric business (retail) and aws inherited the ethos of that retail business. It’s technology is marginally less sophisticated than googles, but googles technology comes with an embarrassingly large amount of smugness and clever solutions that fail in clever ways. Amazon basically just wraps up common technologies in a managed control plane (aside from a few primitives like kinesis, SQS, sns, and DynamoDB) and doesn’t infect everything with their smartness.
I’m here in UK, my bill is sub £10k/mo and literally today few hours ago I had an hour with Google engineer discussing use of their MultiClusterIngress to mix GKE standard and Autopilot to handle spikes extremely fast. Afterwards dude sent me docs and his project he used as an example. It’s the second time this month: we don’t even chat about any specific bugs, more about my architecture goals.
I also have an account manager and she’s currently working on making discount rates from our contract to work with our new hierarchy of multiple billing accounts, some VAT paying and some not, that my finance department requested.
You also can have backup payment methods? I think we have DDs as a default and then CCs as backups.
I was tempted to use Firebase for a new project because it looked so easy. I’m fairly fluent in Angular and it integrates extremely smoothly, so at a surface level it would be the perfect choice for the backend.
Stories like yours – thank you for sharing – plus horror reports about sudden 20k bills have prevented me from taking the “cloud” leap. I’m very glad I didn’t, especially because I also learned lots by setting up a web server and the application layer myself.
I have had GSuite with my domain for 10 years, so theres a lot of emails and apps that I used with Google auth. I always worry about what might happen if something out of my control on Google's end locked me out.
You should backup all of your data and ensure that your domain name is paid well in advance and that you can login to your domain name services w/o 2fa hitting the domain in question.
You can always point your domain to a different provider and reset if you can recv email.
> I am not sure how I could replace Google Login if I point to a different provider.
You can’t. You’ll have to rely on the account recovery mechanisms of the services you used Google login with. Some might be able to switch over to email login after verification, some might not even have a manual way to recover your account / switch it to another login method.
By convenience, backups are often stored on the cloud provider. Yes, this violates the rule of 3, but cloud providers often don't make it easy to do otherwise.
I hear about this kind of thing, all the time. I guess they manage to not do it that often, so I guess I hear only the bad reports, but they are universally awful, when I do hear them, and they all have the same theme:
A.I. Broke it, and the [rare] humans in the loop are powerless to mitigate the A.I.'s breakage.
I was considering using Firebase, when I started the project that I'm working on, now, but got cold feet, and wrote my own server. It took a while, but I'm glad that I did.
There are horror stories like this with all of Big Tech, but Google is hands down the worst offender. I do my best to avoid Google in every corner of my life: the only services I still use are Gmail (slowly migrating to my own domain with another provider though) and Search. But I’ll never use a new Google product anymore because of their unreliability on the long term.
I had this once with Google Drive, when I forgot to enter my new credit card because the existing one had expired. Took weeks to get my files back. It's like people at Google just wait for shit to happen instead of trying to think of the obvious things that could go wrong beforehand.
That being said, if Google is looking for people to lay off there's some obvious candidates. I would include their mangaers too.
I just want to note that I was able to solve this issue. Thank you to the googler who reached out to me to help, 'rygar', and the rest of the google cloud/pay support team for helping fix this issue.
I too am currently in Google Pay support hell. Never in my life have I experienced such a Byzantine hell of weaponized incompetence.
I’d post the thread but it’s just 48 exchanges of me calmly and patiently explaining the situation. And them failing to read the case notes, saying platitudes and then pasting the same non-useful canned response.
Here’s a fun fact or two:
Did you know there’s no way to get your money out from the web interface? Did you know there’s no way to select your profile id on iOS? These two features haven’t been implemented. If you’re clever you can reconstruct the rest. Their documentation indicates that they also don’t have an account closure reconciliation process. I’m hoping that’s just inaccurate documentation because that’d be illegal.
Good times. Some states attorney generals should band together and sue those jerks.
I’m £7.5k/mo in UK: have a named account manager, and an engineer who spends 2h/mo discussing whatever I want on a Google Meet, and bugs I can email directly. Got invited to their Leeds office to say hi and spend a day with them.
> I feel like I'm a man with literally nothing to lose and I can't live with myself anymore.
Dude, I know the pain, but it's only money. Life goes on. If anything, now it should be your mission to stick it to the faceless corporation, not to give in to dark thoughts.
It’s not AI. It’s a jealous contract worker in India or a jealous Google employee because you were making money with your app.
We really need to stop blaming a “Google AI”. Google has 118,000 employees. Someone at Google could help you if they wanted to, but they don’t want to.
I don't like Amazon as a company, but AWS support is great. A couple of times in the past I've made some complaint about them on Twitter, and a rep has actually reached out and helped resolve the issue. And I'm a tiny fish, with monthly bills under $10.
No one from Google ever turns up here and says they care and it matters.
It's now broadly known that developers and businesses don't trust Google, but strangely as far as I can tell, the google CEO has never been held to account for this.
Thank you for reminding me why I avoid Google services like the plague. Even relying on unmanaged apps on a single pet server is less risky than relying on Google.
It's like playing russian roulette with your infra. Google Roulette.
Same happened to me too. They suspended the payment account. Two accounts actually, because different cards. Sent the driver license picture a dozen times, for one account someone eventually took a look and unlocked it (which took about a week). But another one is in review for several months already. And it's impossible to do anything with that.
I see many blame OP for relying on one single cloud provider without having backup and exit strategy. What if she/he is a single persone doing everything and having just barely the time to run the business?
It's the same mentality I have seen from people complaining about Phoronix quality, ignoring It's a one man show website.
I think this is to be expected(sooner or later) for small businesses dealing with "big tech".
You are just a small number in an ocean of "users" so suspending you is really no different than suspending your twitter account. Start taking some responsability folks! Let's stop feeding these walled gardens!
I'm so sorry this happened to you this is why is important to not have a single point of failure. Have domain name in X company, hosting in another, etc.
It seems like a large overhead, but making your app multi-cloud compatible to counter for scenarios such as this might be not be as crazy as it sounds.
99.99% availability doesn't cover for scenarios like this.
That's short-sighted and poor engineering. You need your service to be able to keep going no matter what any single vendor does. Never intentionally create a single point of failure like that.
What are the alternatives? Is Bare Metal with OpenStack or K8s any more resilient to administrative DoS? Do we simply not care about this problem because the businesses being suspended are small fish?
Conversation in this user's comment history. Sounds like he knew the risk he was taking and knew what he'd need to do if it came to that.
-----
alirsgp on Oct 24, 2021 | parent | context | prev | next [–] | on: 5 side projects in 6 years, earning $0
I've published ~20 iOS apps. Only 4 make money, but they allowed me to reach financial independence and never have to work for someone gain at the age of 24.
krater23 on Oct 25, 2021 | parent | prev | next [–]
What's when your apps are removed from Apple for any reason? Do you have enough money to live without this income? Or do you need then just start over again?
alirsgp on Oct 26, 2021 | root | parent | prev | next [–]
I'd have ~2 years and not hurting my lifestyle. Hopefully it doesn't come to that, but I can get a job if needed
I am very sad to hear that this happened to you. It must’ve been absolutely maddening to be unable to talk to a human being as your business was destroyed.
I haven’t had anxiety like this ever before. I’ve worked all day to contact as many ppl as I could. I wish Google would separate their billing from Google Pay and not have this dependency.
GCP like all Google services force you to use Google pay. I have 4 different payment options there, but they are suspending me until they verify my ID. This is a big problem in cloud services since being down for days or weeks or longer can destroy apps and their standings in the app stores.
I want to know what the ban was for. What if the ban was due to a crypto mining operation? All I am asking for is a little more context before pointing any fingers.
Google Cloud CEO Thomas Kurian said "We need scale to be profitable." No, you need to stop treating your customers like crap to be profitable. We don't trust you, you are a pain in our asses, and we have alternatives, so there is really no need for us to spend our money on you. It's now a matter of when we all move to AWS and Azure, not if.
Best of luck!
nope. made the DNS change. went through the motions. even got at call - at 5AM Eastern Time - from a support person to ask me one question, then "I need to confer with so and so". several emails later, new support people asking the same questions. but because I cannot answer "recovery questions" made by person who has left the company, they refuse to offer any other support. attestations of proof of who we are be damned, they just follow their robotic script. so we're just screwed.
Hire a lawyer and send a certified letter with return receipt requiring them to explain their decision, and prepare to sue. I'm presuming that this is a B2B dispute, and I notice that a lot of HN people are (rather counterintuitively, considering Y Combinator owns HN) scared of hiring lawyers. Unless that the records show that your old employee is the recorded administrative owner of the domains (this unfortunately happens frequently to small businesses), this should resolve this issue.
If you really think that the "grass is greener on the other side"...
No, the real solution is to stay away from Big Tech and cloud in general.
Now with Azure, basically they’re making that available to smaller scale players as well.
Meanwhile Google operate GCP in fundamentally the same way they operate free Gmail accounts: hey, this stuff is super neat! So neat actually you bunch of freeloaders are not allowed to complain if this free and neat thing spectacularly breaks once in a while! Wait…you pay for this stuff? Naaah!
They have human support and have proper Enterprise Agreements that don’t turn off your business because some heuristic tripped.
You have to get enough traction on social media to get on someone important at the company’s feed
A false negative security alert from 6 months earlier was the cause, which had blocked CC changes. It took months to resolve but it was manageable.
I know from first hand from a different experience that you can be months overdue with tens of thousands of missing payments on AWS, and tens of thousands in recurring costs without them closing your account. You can effectively run a quarter of credit line from your hosting costs without any issue if you need that.
Your account manager doesn’t even start raising it personally for the first 3 months, they are very easy to deal with.
Google is big. As are Microsoft and Amazon. Google has bad customer service, therefore so do Microsoft and Amazon.
You tried to x in the cloud. You had a bad experience. Therefore the cloud is bad.
In reality of course big companies are not the same. Bigness does not imply sameness. Google support is well known to be rubbish. Microsoft and Amazon support are well known to be excellent. If support matters to you (like when things go wrong) then take that into consideration.
Same for the cloud. There are advantages and disadvantages sure, but providers are different. Google likes to kill services. Others don't. Startups vanish as often as they appear.
Tech is not the only place where you see this false equivalence. All politicians are the same. All fast-food is equally crap. All cars have the same gas mileage. Turns out, no, they are very different, but you need to dig deeper to understand their differences.
Google's aversion to customer service makes them extremely dangerous as a cloud provider in my mind. This also goes for other critical business services, like Gsuite (I know! It's convenient).
If you have GCP or Gsuite now and you're trying to evaluate how big of a deal it is my suggestion would be to pick up a phone and attempt to talk with someone at Google about your account. This experience can act as a preview of what that process might look like when services are turned off.
If you try to call Amazon on the other hand it feels like Jeff Bezos might hop on the call if things aren't going well.
And I've talked to a support human at Google before, about our GSuite.
There are some things which AWS suck at unless you have an enterprise support plan, like having a deep review of a technical issue in their products, and having visibility into product bugs in general.
Even then after reporting a significant issue like Aurora query execution being non-deterministic in a very specific case, I only found out it was solved months later after working around it from reading patch notes.
Google banned my account while I was doing one tutorial. One step failed, the provisioning of something, and I had to redo a few things again. It succeeded but I had used more egress bandwidth than allowed to run the tutorial correctly so they banned me automatically a few minutes later.
The support unbanned me after some time but I thought it was a sign to not use GCP yet.
You won't be the last person to get screwed over by the callous machinery of dehumanised big-tech (AWS or Microsoft would be no better, sadly I think you are rationalising loss). We must all wake up to just how horribly precarious the lives of ordinary people are against the arbitrary whim of opaque, unassailable power centres that masquerade as "Free services".
A friend of mine had the same experience with them.
I was testing software for the upcoming macOS Ventura release, and I only needed 6 hours of initial usage, and then 6 further hours a few weeks later (when a new developer beta dropped) but they closed down my account on the second day.
AWS had no trouble taking my money, so with some finagling I was able to use them instead: https://github.com/thenickdude/proxmox-on-ec2
I used to work for Google and then left for a startup; the sales people kept calling and saying "Urs wants to talk to the CEO" and I'd say: "the CEO hired me to negotiate our cloud deal. We considered GCP but AWS made a far better offer. Urs can call me and we can discuss the limitations of GCP". (I know Urs; I'm the one on stage with him when GCE was launched, running the demo). He never did call and we never gave them our business.
(BTW, my first question for Urs was going to be "why did you defend killing Google Reader? That strongly affected my interest in buying Google products." :)
The TSE should have been able to override the accruing charges on their GCP account. Or they can use another payment method to pay the GCP bill. It sounds like funds they are owed, that they would use to pay GCP bill are locked away until the account is verified.
Very strange.
You'd have to do something like construe the provision of service as an implied contract to continuity in the same way that T&Cs assert that usage of a system implies acceptance of some rules.
I can't remember the number of times they've shut down someone's production workload without notice or warning, gave them no way to resolve/rectify any supposed infraction, and gave no way to contact them to appeal the decision.
I once had to resort to numbering my points and just referring the boilerplate responses (few and far inbetween as they were) back to the numbers over and over until I finally gave up, moved away and never looked back.
In my experience AWS has much better support. Google is famously indifferent to user support issues.
Of course they are. They have the BEST engineers on the planet! How could they ever be wrong? The produ... the customer is obviously wrong in a way they're too dumb to understand.
I had a rudimentary batch job system running. Since I was charged for each power event, I wanted to minimize the power state changes and just queued up the work that needed GPU acceleration until there was enough to justify the cost of at least an entire hour and two power state events. When the cheap, small, always on instance received all the work output from the giant beast server, it would then send a power down command using the aws cli. Yeah, like I said, I was lazy so I just sent off the request and small server just went on about it’s business. Oops.
Like I told the aws billing rep, it would have bankrupted me, ending that venture. I asked that, since it is all virtualized, that while yes I had the virtual hardware provisioned for all that time, if they could get someone to look at the actual utilization they would see it was essentially none. They got back to me and in a day or so to let me know they had forgiven the entire amount.
AWS support actually listen, do real work, and very often it seems, do what is in the best interest of the customer even as it works directly against the company’s bottom line.
I’ve been able to reduce reserved instances when the market has turned unpredictable.
AWS isn’t a pushover, not even a little bit. Dealing with them externally has just been a good experience with professional understanding and a flexible business sense.
Well... AWS has better support IMO (like actual people you can talk to).
But I still chose to not tie myself to any of these platforms for anything critical. It requires additional upfront effort, but I can migrate all my applications off of AWS in a moment.
There is no easy way around Google and Apple when it comes to your app staying available, but I am trying to minimise any chances I could be flagged for anything (basically not linking those accounts to any other services).
Then you are probably not doing "cloud native" development using their proprietary SDKs, so probably just running some EC2 (virtual machine) instance and RDS (database) instance. That is about the most expensive way you can use the big-tech clouds, at that point you might as well move to the smaller hosting providers like Linode.
But I can structure my application so that the innards are not tied to AWS APIs.
For example, SNS is just a small messaging module that I can easily replace for something else by reimplementing literally half a dozen functions.
Guys, these problems have been present in software development for the past half century. They have solutions if you care to find them.
Even if you abstract away most of the platform specific stuff in your code, that's going to take days/weeks of implementation and testing before you can go live. That won't help you when a provider suddenly bans your account in the middle of the night, and you need it running asap.
Having a couple of days of downtime might be an acceptable tradeoff for an event with a very low chance of happening. Risk management basically. (If your business doesn’t survive that downtime it might be a completely different story of course.)
My CEO is constantly beating it into my head: "I will gladly accept a day or even more of downtime from time to time if it lets me get what I want 10% faster".
It does not mean shoddy engineering. It just means consciously deciding not to do some work. Spending couple more days to get the app automatically fail over to another without downtime? Engineer it to run as a distributed system when it could happily run on a single server? Let's just not do it, let's make it simpler and easier to make and accept that if AWS has a failure it may take a moment to spin another instance somewhere else.
GCP is not going to be a thing in a few years. Why would anyone pick GCP at this point with all the horror stories coming out of there? Yes, support is hard, yes it's expensive, yes it takes time. But at the end of the day this is how you make or break the trust of your customers.
It's not uncommon to have these same small to medium sized business owners ask for help with other suspended accounts in other Google apps like GoogleAds or Google Merchant Center.
The support across all platforms is a growing eyesore for Google, and at this point I hope becomes the catalyst for real change in this space.
It was bad enough when Yelp would act like the Mafia in regards to how they approached and treated businesses, but this level of negligence (from such a tech giant) is something to really see.
We've been advocating on behalf of all these small businesses for years and truely hope that the outcry grows to a fever pitch where no more band aid solutions are used anymore.
> “You've got an additional problem though, which is that this tells us you have two support channels: one that doesn't work (i.e. yours, the one you built), and one that does (Twitter-shaming). The first channel represents how you act when no one's watching; the second, how you act when they are. Most people prefer to deal with people for whom those two are the same.”
From: https://news.ycombinator.com/item?id=20066239
I ended up calling an ex-coworker who still worked at google. He sent an internal email. It was resolved the same day. I never found out what the issue was.
Don’t use google for anything you care about, unless you maintain close friendships with googlers.
Over the years I asked two different friends at Google for help, one of whom was management. They were baffled when the internal message thing yielded no results either. There doesn't seem to be any process for getting certain types of issues resolved at google.
My point is that google can ruin people's lives even if the victims aren't the ones using it. Gmail blocking inbound mail is another example - if gmail decides to block your service, everyone will blame you, and there's nothing you can do about it other than wait.
And as a customer, I never want to be in a position where I ever see $megabill. Maybe I might be able to get reversed if the gods smile upon me that day, but it would be incredibly stressful until I had resolutions. Seems an obvious blindside for “customer obsessed”. Anything I might host is practically a joke for which I would gladly have the provider pull the plug if billing suddenly spiked above pre-established threshold. I’ll take the downtime if it helps me sleep at night.
I would imagine AWSs profit is in servicing large clients using lots of CPU/bandwidth, so the only reason to even bother with small clients is some kind of low-probability funnel for the occasional startup that blows up. Or just a loss-leader strategy for market domination (but only if that small-biz experience is vastly superior to other clouds' offering).
One way or another, small businesses should never trust that big businesses are going to look out for them, for any number of "rational" reasons on the big biz side, especially if it's not clear you're their bread-and-butter.
I don’t understand why anyone would ever choose them for anything critical.
Free support for AWS is better than anything Google has (at least based on my reading, for obvious reasons I have no first-hand experience).
I just can’t fathom Google’s inability to sort this out. They’ve been the laughing stock of serious cloud business for years.
1. SaaSy kids in SV or who wish they were in SV who also think Google Sheets beats Excel.
2. Big enterprise executives who heard Google was what the cool SaaSy kids do from someone in I.T. then got lured by too-good-to-be-true discounts from Google Cloud sales team as long as the executive is happy to be featured for their forward-thinking tech chops in pressers about picking GCP. To their credit, they will undo the Office to Workspace migration mandate almost immediately, after making their 140,000 users attempt it from IE 10 on Win 7, but there won't be press about that.
I loled out loud
are you supposed to write your own firebase? develop your own android? yes, OP learned a hard lesson about Google, but that does not mean that you cannot build stuff using just one provider. you need to do your due diligence of course. You need to have a plan, of course. But the cost of doing this on 2 platforms skyrocket. And especially for a small player the multi-cloud, massive investment is just not an option.
Are new devs taught that having something in two AWS zones equals redundancy? You drank the Kool-aid, people.
I would port your stuff to aws. Most services have close equivalents and it is probably worth while to try. As another poster noted it might be useful to have cross provider redundancy.
I’ll be honest I have refused to work at gcp and have steered the mega corps I’ve advised on cloud choices explicitly away from gcp. Azure is better, but less technically sophisticated. It’s still a one shop cloud. And any company whose core business is surveillance and advertising won’t care about their customers - you’re just meat for their machine. Amazon was built around a customer centric business (retail) and aws inherited the ethos of that retail business. It’s technology is marginally less sophisticated than googles, but googles technology comes with an embarrassingly large amount of smugness and clever solutions that fail in clever ways. Amazon basically just wraps up common technologies in a managed control plane (aside from a few primitives like kinesis, SQS, sns, and DynamoDB) and doesn’t infect everything with their smartness.
Good luck.
For me, the key factor is that there's nobody to call. Want support? You had better be famous on twitter.
I’m here in UK, my bill is sub £10k/mo and literally today few hours ago I had an hour with Google engineer discussing use of their MultiClusterIngress to mix GKE standard and Autopilot to handle spikes extremely fast. Afterwards dude sent me docs and his project he used as an example. It’s the second time this month: we don’t even chat about any specific bugs, more about my architecture goals.
I also have an account manager and she’s currently working on making discount rates from our contract to work with our new hierarchy of multiple billing accounts, some VAT paying and some not, that my finance department requested.
You also can have backup payment methods? I think we have DDs as a default and then CCs as backups.
Stories like yours – thank you for sharing – plus horror reports about sudden 20k bills have prevented me from taking the “cloud” leap. I’m very glad I didn’t, especially because I also learned lots by setting up a web server and the application layer myself.
You can always point your domain to a different provider and reset if you can recv email.
You can’t. You’ll have to rely on the account recovery mechanisms of the services you used Google login with. Some might be able to switch over to email login after verification, some might not even have a manual way to recover your account / switch it to another login method.
Not being able to survive a cloud vendor cancelling you is the 21st century equivalent of not having any backups.
Being brought down by a single vendor is poor engineering.
A.I. Broke it, and the [rare] humans in the loop are powerless to mitigate the A.I.'s breakage.
I was considering using Firebase, when I started the project that I'm working on, now, but got cold feet, and wrote my own server. It took a while, but I'm glad that I did.
That being said, if Google is looking for people to lay off there's some obvious candidates. I would include their mangaers too.
Disclaimer: Googler, not in this area.
Disclaimer: Xoogler, who worked on Dasher and other paying-customer-facing tools.
I’d post the thread but it’s just 48 exchanges of me calmly and patiently explaining the situation. And them failing to read the case notes, saying platitudes and then pasting the same non-useful canned response.
Here’s a fun fact or two: Did you know there’s no way to get your money out from the web interface? Did you know there’s no way to select your profile id on iOS? These two features haven’t been implemented. If you’re clever you can reconstruct the rest. Their documentation indicates that they also don’t have an account closure reconciliation process. I’m hoping that’s just inaccurate documentation because that’d be illegal.
Good times. Some states attorney generals should band together and sue those jerks.
Don't ever use GCP if you're not a big enough enterprise. Just as you should never buy a Dell laptop as an individual consumer.
I'm sorry this happened to you, I hope someone will learn from your experience
Dude, I know the pain, but it's only money. Life goes on. If anything, now it should be your mission to stick it to the faceless corporation, not to give in to dark thoughts.
We really need to stop blaming a “Google AI”. Google has 118,000 employees. Someone at Google could help you if they wanted to, but they don’t want to.
I don't like Amazon as a company, but AWS support is great. A couple of times in the past I've made some complaint about them on Twitter, and a rep has actually reached out and helped resolve the issue. And I'm a tiny fish, with monthly bills under $10.
It's now broadly known that developers and businesses don't trust Google, but strangely as far as I can tell, the google CEO has never been held to account for this.
It's like playing russian roulette with your infra. Google Roulette.
You are just a small number in an ocean of "users" so suspending you is really no different than suspending your twitter account. Start taking some responsability folks! Let's stop feeding these walled gardens!
That way if one turns crazy there's More hope
My business uses Firebase, and I'm having serious second thoughts about that right now.
-----
alirsgp on Oct 24, 2021 | parent | context | prev | next [–] | on: 5 side projects in 6 years, earning $0
I've published ~20 iOS apps. Only 4 make money, but they allowed me to reach financial independence and never have to work for someone gain at the age of 24.
krater23 on Oct 25, 2021 | parent | prev | next [–]
What's when your apps are removed from Apple for any reason? Do you have enough money to live without this income? Or do you need then just start over again?
alirsgp on Oct 26, 2021 | root | parent | prev | next [–]
I'd have ~2 years and not hurting my lifestyle. Hopefully it doesn't come to that, but I can get a job if needed
geez guys keep your business away from google, it's tough not being able to reach a damn human for support especially when you're a paying one.
Having a service that is getting network traffic for both crypto related things and 18+ dating may have raised some algorithmic alarm bells.