Long story short, Google froze my Google Pay account which is used to pay for anything related to Google services, including GCP. This was done as they need to temporarily suspend me to 'verify' me. The GCP team couldn't help me since it was an issue with Google Pay. Not to mention their chat support is useless and doesn't have any power to look into cases.
Now I login to find out my Google Pay account is 'temporarily suspended' since I need to verify my account. I go ahead and add my drivers license and fill out their form, but while they verify details they suspend your ability to pay for any Google services.
So now I can't pay for GCP usage (firebase) and all my authentication, database, cloud functions, are all not usable since the service is suspended. Unlike Amazon who offers phone lines for resolving billing issues, Google does no such thing. I wish I chose AWS.
The end result is that I have apps I spent years of my life building and which currently feed me to only be rendered unusable because some AI at google thought my Google Pay needed additional verification.
My whole life has done a 180 in the past week. I feel like I'm a man with literally nothing to lose and I can't live with myself anymore. This hurts much more than losing a job for me.
Please stay far, far away from Google services of any kind. They are by far the worst of the big tech companies when it comes to developer support.
Google Cloud CEO Thomas Kurian said "We need scale to be profitable." No, you need to stop treating your customers like crap to be profitable. We don't trust you, you are a pain in our asses, and we have alternatives, so there is really no need for us to spend our money on you. It's now a matter of when we all move to AWS and Azure, not if.
Best of luck!
nope. made the DNS change. went through the motions. even got at call - at 5AM Eastern Time - from a support person to ask me one question, then "I need to confer with so and so". several emails later, new support people asking the same questions. but because I cannot answer "recovery questions" made by person who has left the company, they refuse to offer any other support. attestations of proof of who we are be damned, they just follow their robotic script. so we're just screwed.
Hire a lawyer and send a certified letter with return receipt requiring them to explain their decision, and prepare to sue. I'm presuming that this is a B2B dispute, and I notice that a lot of HN people are (rather counterintuitively, considering Y Combinator owns HN) scared of hiring lawyers. Unless that the records show that your old employee is the recorded administrative owner of the domains (this unfortunately happens frequently to small businesses), this should resolve this issue.
If you really think that the "grass is greener on the other side"...
No, the real solution is to stay away from Big Tech and cloud in general.
Now with Azure, basically they’re making that available to smaller scale players as well.
Meanwhile Google operate GCP in fundamentally the same way they operate free Gmail accounts: hey, this stuff is super neat! So neat actually you bunch of freeloaders are not allowed to complain if this free and neat thing spectacularly breaks once in a while! Wait…you pay for this stuff? Naaah!
They have human support and have proper Enterprise Agreements that don’t turn off your business because some heuristic tripped.
You have to get enough traction on social media to get on someone important at the company’s feed
A false negative security alert from 6 months earlier was the cause, which had blocked CC changes. It took months to resolve but it was manageable.
I know from first hand from a different experience that you can be months overdue with tens of thousands of missing payments on AWS, and tens of thousands in recurring costs without them closing your account. You can effectively run a quarter of credit line from your hosting costs without any issue if you need that.
Your account manager doesn’t even start raising it personally for the first 3 months, they are very easy to deal with.
Google is big. As are Microsoft and Amazon. Google has bad customer service, therefore so do Microsoft and Amazon.
You tried to x in the cloud. You had a bad experience. Therefore the cloud is bad.
In reality of course big companies are not the same. Bigness does not imply sameness. Google support is well known to be rubbish. Microsoft and Amazon support are well known to be excellent. If support matters to you (like when things go wrong) then take that into consideration.
Same for the cloud. There are advantages and disadvantages sure, but providers are different. Google likes to kill services. Others don't. Startups vanish as often as they appear.
Tech is not the only place where you see this false equivalence. All politicians are the same. All fast-food is equally crap. All cars have the same gas mileage. Turns out, no, they are very different, but you need to dig deeper to understand their differences.
Google's aversion to customer service makes them extremely dangerous as a cloud provider in my mind. This also goes for other critical business services, like Gsuite (I know! It's convenient).
If you have GCP or Gsuite now and you're trying to evaluate how big of a deal it is my suggestion would be to pick up a phone and attempt to talk with someone at Google about your account. This experience can act as a preview of what that process might look like when services are turned off.
If you try to call Amazon on the other hand it feels like Jeff Bezos might hop on the call if things aren't going well.
And I've talked to a support human at Google before, about our GSuite.
There are some things which AWS suck at unless you have an enterprise support plan, like having a deep review of a technical issue in their products, and having visibility into product bugs in general.
Even then after reporting a significant issue like Aurora query execution being non-deterministic in a very specific case, I only found out it was solved months later after working around it from reading patch notes.
Google banned my account while I was doing one tutorial. One step failed, the provisioning of something, and I had to redo a few things again. It succeeded but I had used more egress bandwidth than allowed to run the tutorial correctly so they banned me automatically a few minutes later.
The support unbanned me after some time but I thought it was a sign to not use GCP yet.
You won't be the last person to get screwed over by the callous machinery of dehumanised big-tech (AWS or Microsoft would be no better, sadly I think you are rationalising loss). We must all wake up to just how horribly precarious the lives of ordinary people are against the arbitrary whim of opaque, unassailable power centres that masquerade as "Free services".
A friend of mine had the same experience with them.
I was testing software for the upcoming macOS Ventura release, and I only needed 6 hours of initial usage, and then 6 further hours a few weeks later (when a new developer beta dropped) but they closed down my account on the second day.
AWS had no trouble taking my money, so with some finagling I was able to use them instead: https://github.com/thenickdude/proxmox-on-ec2
I used to work for Google and then left for a startup; the sales people kept calling and saying "Urs wants to talk to the CEO" and I'd say: "the CEO hired me to negotiate our cloud deal. We considered GCP but AWS made a far better offer. Urs can call me and we can discuss the limitations of GCP". (I know Urs; I'm the one on stage with him when GCE was launched, running the demo). He never did call and we never gave them our business.
(BTW, my first question for Urs was going to be "why did you defend killing Google Reader? That strongly affected my interest in buying Google products." :)
The TSE should have been able to override the accruing charges on their GCP account. Or they can use another payment method to pay the GCP bill. It sounds like funds they are owed, that they would use to pay GCP bill are locked away until the account is verified.
Very strange.
You'd have to do something like construe the provision of service as an implied contract to continuity in the same way that T&Cs assert that usage of a system implies acceptance of some rules.
I can't remember the number of times they've shut down someone's production workload without notice or warning, gave them no way to resolve/rectify any supposed infraction, and gave no way to contact them to appeal the decision.
I once had to resort to numbering my points and just referring the boilerplate responses (few and far inbetween as they were) back to the numbers over and over until I finally gave up, moved away and never looked back.
In my experience AWS has much better support. Google is famously indifferent to user support issues.
Of course they are. They have the BEST engineers on the planet! How could they ever be wrong? The produ... the customer is obviously wrong in a way they're too dumb to understand.
I had a rudimentary batch job system running. Since I was charged for each power event, I wanted to minimize the power state changes and just queued up the work that needed GPU acceleration until there was enough to justify the cost of at least an entire hour and two power state events. When the cheap, small, always on instance received all the work output from the giant beast server, it would then send a power down command using the aws cli. Yeah, like I said, I was lazy so I just sent off the request and small server just went on about it’s business. Oops.
Like I told the aws billing rep, it would have bankrupted me, ending that venture. I asked that, since it is all virtualized, that while yes I had the virtual hardware provisioned for all that time, if they could get someone to look at the actual utilization they would see it was essentially none. They got back to me and in a day or so to let me know they had forgiven the entire amount.
AWS support actually listen, do real work, and very often it seems, do what is in the best interest of the customer even as it works directly against the company’s bottom line.
I’ve been able to reduce reserved instances when the market has turned unpredictable.
AWS isn’t a pushover, not even a little bit. Dealing with them externally has just been a good experience with professional understanding and a flexible business sense.
Well... AWS has better support IMO (like actual people you can talk to).
But I still chose to not tie myself to any of these platforms for anything critical. It requires additional upfront effort, but I can migrate all my applications off of AWS in a moment.
There is no easy way around Google and Apple when it comes to your app staying available, but I am trying to minimise any chances I could be flagged for anything (basically not linking those accounts to any other services).
Then you are probably not doing "cloud native" development using their proprietary SDKs, so probably just running some EC2 (virtual machine) instance and RDS (database) instance. That is about the most expensive way you can use the big-tech clouds, at that point you might as well move to the smaller hosting providers like Linode.
But I can structure my application so that the innards are not tied to AWS APIs.
For example, SNS is just a small messaging module that I can easily replace for something else by reimplementing literally half a dozen functions.
Guys, these problems have been present in software development for the past half century. They have solutions if you care to find them.
Even if you abstract away most of the platform specific stuff in your code, that's going to take days/weeks of implementation and testing before you can go live. That won't help you when a provider suddenly bans your account in the middle of the night, and you need it running asap.
Having a couple of days of downtime might be an acceptable tradeoff for an event with a very low chance of happening. Risk management basically. (If your business doesn’t survive that downtime it might be a completely different story of course.)
My CEO is constantly beating it into my head: "I will gladly accept a day or even more of downtime from time to time if it lets me get what I want 10% faster".
It does not mean shoddy engineering. It just means consciously deciding not to do some work. Spending couple more days to get the app automatically fail over to another without downtime? Engineer it to run as a distributed system when it could happily run on a single server? Let's just not do it, let's make it simpler and easier to make and accept that if AWS has a failure it may take a moment to spin another instance somewhere else.
GCP is not going to be a thing in a few years. Why would anyone pick GCP at this point with all the horror stories coming out of there? Yes, support is hard, yes it's expensive, yes it takes time. But at the end of the day this is how you make or break the trust of your customers.
It's not uncommon to have these same small to medium sized business owners ask for help with other suspended accounts in other Google apps like GoogleAds or Google Merchant Center.
The support across all platforms is a growing eyesore for Google, and at this point I hope becomes the catalyst for real change in this space.
It was bad enough when Yelp would act like the Mafia in regards to how they approached and treated businesses, but this level of negligence (from such a tech giant) is something to really see.
We've been advocating on behalf of all these small businesses for years and truely hope that the outcry grows to a fever pitch where no more band aid solutions are used anymore.
> “You've got an additional problem though, which is that this tells us you have two support channels: one that doesn't work (i.e. yours, the one you built), and one that does (Twitter-shaming). The first channel represents how you act when no one's watching; the second, how you act when they are. Most people prefer to deal with people for whom those two are the same.”
From: https://news.ycombinator.com/item?id=20066239
I ended up calling an ex-coworker who still worked at google. He sent an internal email. It was resolved the same day. I never found out what the issue was.
Don’t use google for anything you care about, unless you maintain close friendships with googlers.
Over the years I asked two different friends at Google for help, one of whom was management. They were baffled when the internal message thing yielded no results either. There doesn't seem to be any process for getting certain types of issues resolved at google.
My point is that google can ruin people's lives even if the victims aren't the ones using it. Gmail blocking inbound mail is another example - if gmail decides to block your service, everyone will blame you, and there's nothing you can do about it other than wait.
And as a customer, I never want to be in a position where I ever see $megabill. Maybe I might be able to get reversed if the gods smile upon me that day, but it would be incredibly stressful until I had resolutions. Seems an obvious blindside for “customer obsessed”. Anything I might host is practically a joke for which I would gladly have the provider pull the plug if billing suddenly spiked above pre-established threshold. I’ll take the downtime if it helps me sleep at night.
I would imagine AWSs profit is in servicing large clients using lots of CPU/bandwidth, so the only reason to even bother with small clients is some kind of low-probability funnel for the occasional startup that blows up. Or just a loss-leader strategy for market domination (but only if that small-biz experience is vastly superior to other clouds' offering).
One way or another, small businesses should never trust that big businesses are going to look out for them, for any number of "rational" reasons on the big biz side, especially if it's not clear you're their bread-and-butter.
I don’t understand why anyone would ever choose them for anything critical.
Free support for AWS is better than anything Google has (at least based on my reading, for obvious reasons I have no first-hand experience).
I just can’t fathom Google’s inability to sort this out. They’ve been the laughing stock of serious cloud business for years.
1. SaaSy kids in SV or who wish they were in SV who also think Google Sheets beats Excel.
2. Big enterprise executives who heard Google was what the cool SaaSy kids do from someone in I.T. then got lured by too-good-to-be-true discounts from Google Cloud sales team as long as the executive is happy to be featured for their forward-thinking tech chops in pressers about picking GCP. To their credit, they will undo the Office to Workspace migration mandate almost immediately, after making their 140,000 users attempt it from IE 10 on Win 7, but there won't be press about that.
I loled out loud
are you supposed to write your own firebase? develop your own android? yes, OP learned a hard lesson about Google, but that does not mean that you cannot build stuff using just one provider. you need to do your due diligence of course. You need to have a plan, of course. But the cost of doing this on 2 platforms skyrocket. And especially for a small player the multi-cloud, massive investment is just not an option.
Are new devs taught that having something in two AWS zones equals redundancy? You drank the Kool-aid, people.
I would port your stuff to aws. Most services have close equivalents and it is probably worth while to try. As another poster noted it might be useful to have cross provider redundancy.
I’ll be honest I have refused to work at gcp and have steered the mega corps I’ve advised on cloud choices explicitly away from gcp. Azure is better, but less technically sophisticated. It’s still a one shop cloud. And any company whose core business is surveillance and advertising won’t care about their customers - you’re just meat for their machine. Amazon was built around a customer centric business (retail) and aws inherited the ethos of that retail business. It’s technology is marginally less sophisticated than googles, but googles technology comes with an embarrassingly large amount of smugness and clever solutions that fail in clever ways. Amazon basically just wraps up common technologies in a managed control plane (aside from a few primitives like kinesis, SQS, sns, and DynamoDB) and doesn’t infect everything with their smartness.
Good luck.
For me, the key factor is that there's nobody to call. Want support? You had better be famous on twitter.
You can always point your domain to a different provider and reset if you can recv email.
You can’t. You’ll have to rely on the account recovery mechanisms of the services you used Google login with. Some might be able to switch over to email login after verification, some might not even have a manual way to recover your account / switch it to another login method.
Not being able to survive a cloud vendor cancelling you is the 21st century equivalent of not having any backups.
Being brought down by a single vendor is poor engineering.
A.I. Broke it, and the [rare] humans in the loop are powerless to mitigate the A.I.'s breakage.
I was considering using Firebase, when I started the project that I'm working on, now, but got cold feet, and wrote my own server. It took a while, but I'm glad that I did.
I’m here in UK, my bill is sub £10k/mo and literally today few hours ago I had an hour with Google engineer discussing use of their MultiClusterIngress to mix GKE standard and Autopilot to handle spikes extremely fast. Afterwards dude sent me docs and his project he used as an example. It’s the second time this month: we don’t even chat about any specific bugs, more about my architecture goals.
I also have an account manager and she’s currently working on making discount rates from our contract to work with our new hierarchy of multiple billing accounts, some VAT paying and some not, that my finance department requested.
You also can have backup payment methods? I think we have DDs as a default and then CCs as backups.
That being said, if Google is looking for people to lay off there's some obvious candidates. I would include their mangaers too.
Disclaimer: Googler, not in this area.
Disclaimer: Xoogler, who worked on Dasher and other paying-customer-facing tools.
Stories like yours – thank you for sharing – plus horror reports about sudden 20k bills have prevented me from taking the “cloud” leap. I’m very glad I didn’t, especially because I also learned lots by setting up a web server and the application layer myself.
I’d post the thread but it’s just 48 exchanges of me calmly and patiently explaining the situation. And them failing to read the case notes, saying platitudes and then pasting the same non-useful canned response.
Here’s a fun fact or two: Did you know there’s no way to get your money out from the web interface? Did you know there’s no way to select your profile id on iOS? These two features haven’t been implemented. If you’re clever you can reconstruct the rest. Their documentation indicates that they also don’t have an account closure reconciliation process. I’m hoping that’s just inaccurate documentation because that’d be illegal.
Good times. Some states attorney generals should band together and sue those jerks.
Don't ever use GCP if you're not a big enough enterprise. Just as you should never buy a Dell laptop as an individual consumer.
I'm sorry this happened to you, I hope someone will learn from your experience
Dude, I know the pain, but it's only money. Life goes on. If anything, now it should be your mission to stick it to the faceless corporation, not to give in to dark thoughts.
We really need to stop blaming a “Google AI”. Google has 118,000 employees. Someone at Google could help you if they wanted to, but they don’t want to.
I don't like Amazon as a company, but AWS support is great. A couple of times in the past I've made some complaint about them on Twitter, and a rep has actually reached out and helped resolve the issue. And I'm a tiny fish, with monthly bills under $10.
It's now broadly known that developers and businesses don't trust Google, but strangely as far as I can tell, the google CEO has never been held to account for this.
It's like playing russian roulette with your infra. Google Roulette.
You are just a small number in an ocean of "users" so suspending you is really no different than suspending your twitter account. Start taking some responsability folks! Let's stop feeding these walled gardens!
That way if one turns crazy there's More hope
My business uses Firebase, and I'm having serious second thoughts about that right now.
geez guys keep your business away from google, it's tough not being able to reach a damn human for support especially when you're a paying one.
-----
alirsgp on Oct 24, 2021 | parent | context | prev | next [–] | on: 5 side projects in 6 years, earning $0
I've published ~20 iOS apps. Only 4 make money, but they allowed me to reach financial independence and never have to work for someone gain at the age of 24.
krater23 on Oct 25, 2021 | parent | prev | next [–]
What's when your apps are removed from Apple for any reason? Do you have enough money to live without this income? Or do you need then just start over again?
alirsgp on Oct 26, 2021 | root | parent | prev | next [–]
I'd have ~2 years and not hurting my lifestyle. Hopefully it doesn't come to that, but I can get a job if needed
Having a service that is getting network traffic for both crypto related things and 18+ dating may have raised some algorithmic alarm bells.