This type of issue can be incredibly annoying to deal with, because the legitimate answer to the abuse report ("someone is spoofing my IP, it isn't me, and the machine is not compromised") is the exact same excuse that a malicious actor would provide.
Then, as noted in the article, you're trying to prove a negative to someone who doesn't really care at all, which is borderline impossible.
Hertzner says in the email that no response is necessary.
Automated abuse reports of things that are easily spoofed don't justify a report, but might justify a quick check to make sure your box is still operating correctly and hasn't been taken over.
>but we do expect you to check it and to resolve any potential issues.
That's the important part.
If they receive another one (or two, or a few) more abuse reports, they assume it is not fixed, and will expect a response then. Which ends up being annoying.
> the legitimate answer to the abuse report ("someone is spoofing my IP, it isn't me, and the machine is not compromised") is the exact same excuse that a malicious actor would provide.
The legitimate answer would include some sort of real-world attestation about you from a trusted third party. Probably the very least, some evidence of your identity and jurisdiction. Maybe including a video call or something. Not just you anonymously claiming you're a good guy over the internet and expecting to be believed.
Netflow data or equivalent? I'd assume any provider to have such records, at least in the short term. It can also be invaluable in debugging network problems post-hoc.
>The legitimate answer would include some sort of real-world attestation about you from a trusted third party.
It's annoying to find someone (or some service) that is willing to attest on your behalf and have that person (or service) be trusted by your provider more than whoever filed the abuse complaint.
>Maybe including a video call or something.
It's annoying to find someone at your provider who will take the time to do this. It's annoying to take my time to have to do this.
My point, overall, was that this is just a really annoying problem.
So it turns out at the network service level, anonymity has never been guaranteed. If I, as another chunk of the network, can't trust your chunk, it's going to get cut from accessing me.
There has to be some ability to establish baseline trust.
It's true due to the nature of what the network is.
In the abstract: if I own the infrastructure and someone uses that infrastructure to hurt someone, that someone who was hurt (or the parties who protect them) are going to come to me asking questions. If I just say "I don't know" and the law doesn't protect my willful ignorance, I'm at best enabling harm; I'm at worst socially or legally liable for negligence.
In the abstract, the systems of human governance recognize harm and seek to mitigate it.
So if I'm peered to a network using me as a bridge to do harm, I can't trust that network when the bad starts to outweigh the good. If I can't establish trust via human methods, I'm gonna cut that network off to protect myself.
(The Internet started as people who had working relationships with each other and grew out from there. Even though the web of connections is much larger and more indirect now, the whole thing is still at its core a human construct and beholden to human standards of conduct, because humans ultimately have their hands on the various plugs that are yankable).
> The internet was broken 25 years ago and is still broken 25 years later. Spoofed source IP addresses should not still be a problem in 2024, but the larger internet community seems completely unwilling to enforce any kind of rules or baseline security that would make the internet safer for everyone.
Same with spoofed MAC addresses, email addresses, ARP messages, Neighbor Discovery, MitM TLS certificates ... It's amazing anything works anymore :D
The thing is, obviously, that the Internet isn't broken, it has incredible utility and reliability. If it was designed and operated to be perfect, then it would likely be massively broken quite often. It is the tolerance for mild brokenness that has contributed significantly to its robustness and utility.
That isn't an argument for not improving things though, just a warning against perfection, if you chase it then you're liable to make really big mistakes that ruin everything.
It’s quite sad the only mail server out there which checks if you are allowed to use a email address is exchange. With all others you can set the from: header however you like.
Who cares whether it's the MTA that does it or a collection of daemons invoked by the MTA? Just get things configured correctly, and you should be gold.
Now as far as every other mail operator setting up their stuff right such that From spoofing is no longer feasible, well... Can't help ya there. I don't run my email to make money, so the incentive to adopt pathological configs for the sake of maximizing the number of users/Domains who can send from one IP ain't there.
This is nothing new. A few years back, I implemented a very basic firewall rule: if I received a TCP packet with SYN=1 and ACK=0 to destination port 22, the source IP would get blacklisted for a day. But then I started getting complaints about certain sites and services not working. It turned out that every few days, I'd receive such packets from IPs like 8.8.8.8 or 1.1.1.1, as well as from Steam, Roblox, Microsoft, and all kinds of popular servers—Facebook, Instagram, and various chat services. Of course, these were all spoofed packets, which eventually led me to adjust my firewall rules to require a bit more validation.
So, I can assure you this is quite common. As a personal note, I know I’m a bit of an exception for operating multiple IP addresses, but I need the flexibility to send packets with any of my source addresses through any of my ISPs. That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
> As a personal note, I know I’m a bit of an exception for operating multiple IP addresses, but I need the flexibility to send packets with any of my source addresses through any of my ISPs. That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
If you actually have your own IP addresses this is normal and expected, but if you're able to use ISP A's IP addresses through ISP B or vice versa that has always been a bug that you are wrong to use.
If you are doing the latter this is firmly in the "reenable spacebar heating" category and I hope your ISPs fix their broken networks.
Okay, looks like I will reply to a few of the comments to clarify things.
I’ll give a concrete, real example.
I worked at a company that hosted some web assets on-prem in one of their branches. They had a 1Gbps connection there. However, at HQ, we had multiple 10G connections and a pretty good data center. So, we moved the web VM to HQ but kept the assigned IP address (a public static from ISP-A). We routed it through a VPN to HQ. The server used our default GW and sent responses with source IP (ISP-A) via ISP-B (10G).
That way, we utilized 10G outbound, even though the inbound was limited to 1G. It was only for GET requests anyway. I know this wasn’t the most optimal setup, and we eventually changed the IP, but it seems like a valid use case.
Scenario 2: We had two connections from two different ISPs (our own ASN, our own /23 addresses). We wanted to load balance some traffic and sent half of our IPs through ISP-A and the other half through ISP-B. It worked fine, but when we tried to mix the balance a bit, we found an interesting glitch. We announced the first /24 to ISP-A and the second /24 to ISP-B, but ISP-A had RP filtering. So, we had to announce all the IPs to them.
The way the RP filter works, as you may guess, means we cannot prepend or anything. All traffic must come through them. If they see a better route for that prefix, they will filter it. For a few months, they refused to fix this, citing security. There’s no shame in security best practices, so I might as well name the ISP—Virgin Media.
Note that the internet with rp_filter is not $20/month. It was more like 5K+/month!! And we did not change it due to lack of alternatives there. But otherwise guess who loses the contract :)
In your first scenario, any connections established through the ISP-A's IP address would be routed back through the VPN connection that they came in on. If that server were to establish it's own connections to external resources, it would feasibly be able to use the 10g connection from ISP-B. It would not be able to dictate what source address was used with connections coming from ISP-B.
It could work the way OP described if they routed all outbound traffic via ISP-A regardless of source address, and ISP-A allowed spoofing. I think that's what they meant.
It is common practice for business subscribers (around UK) to get a /29
On the router we add a single /32 via the tunnel.
I think even the cheapest 100bucks business plans from many ISPs come with /28 or /29.
It is a complete waste because we had like 10 offices with 3-5 persons with laptops and NO servers. The common question from the ISPs is: Do you need some IPs? When we answer no, they give us /29.
>As a personal note, I know I’m a bit of an exception ...That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
"...and obviously, Pennywise, I must spoof ingress and egress..."
Not really. Early IPv6 documentation kind of assumed that the vast address space would lead towards hierarchical addressing and that a multi-homed user would use addresses assigned by all of their ISPs, but at least in my experience, that doesn't really pan out --- if you have router advertisements from two different ISP prefixes, automatic configuration on common OSes (windows, linux, freebsd) will lead towards often sending traffic with ISP A through the router from ISP B, which doesn't really work well, especially if either or both ISPs run prefix filters. There's probably ways to make that style of multihoming work, but it's not fun.
Turns out, most multiphomed IPv6 users need provider indepdent addresses, just like with IPv4. And then you need to make sure your all your ISPs allow you to use all your prefixes. On the plus side, it's much more likely to get an IPv6 allocation that's contiguous and that you won't outgrow; so probably you only need one v6 prefix, and you may not need to change it as often as with v4.
The advantage of IPv6 is that can multiple addresses. This means that good way to organize network is to have machines use local provider addresses to access the Internet.
Then have ULA addresses for internal network. Those will be routed with tunnels and VPNs. That separates accessing the internet from internal network, and means that don't need to have routable address space.
The only people who would need own address space have data centers and routers.
Yeah, there are ways to make it work, for example by specifying source addresses or nets on the the routes. In openwrt it's a checkbox to tick on the upstream interfaces.
> Which means, if you just find one transit provider which doesn’t do BCP38 filtering… you can send IP packets tagged with any source IP you want! And unfortunately, even though the origins of BCP38 date back to 1998… there are still network providers 25 years later that don’t implement it.
What would it take to get enough network providers to start rejecting traffic from all ASes that don't implement this, so that spoofing was no longer possible?
Cloudflare is probably enough. They already control enough ingress that their "checking the security of your connection" could actually mean something.
You'd have to find some way to make network providers care. Especially 'tier 1' transit providers and other networks of unusual size.
It's much easier to work on reducing reflection multipliers though, because you can scan (ipv4 anyway) for reflection vectors and yell at people that will respond with 10x the input bytes.
The “someone hates Tor relays” theory doesn’t sound worth the effort. This could be an entity running malicious relays, while also trying to unethically take down legitimate relays to increase the percentage of the network that they control.
This is almost certainly it. There’s a lot of head-sand-burying around here about just how easily an attacker with access to logs of a not-even-that-large segment of the nodes can gain visibility into individuals’ service access patterns.
Back in the day I would scan for DrDoS reflectors in a similar way, no hosting provider wants to get reports for port scanning so the source address of the scan would belong to an innocent cloud provider with a reputable IP that reflectors would happily send UDP replies to. The cloud provider would of course get a massive influx of complaints but you would just say that you aren't doing any scanning from your server (which they would verify) and they wouldn't shut your service off. The server sending out the spoofed scan packets is undetectable so you're able to scan the entire internet repeatedly without the typical abuse issues that come with it.
I'm not sure how often this happens in practice but tracing the source of a spoofed packet is possible if you can coordinate with transit providers to follow the hops back to the source. One time JPMorgan worked with Cogent to tell us to stop sending packets with their IP addresses (Cogent is one of the most spoofer friendly tier 1's on the internet btw).
This is the first time I've heard of this being used to target TOR specifically which seems counterintuitive, you would think people sending out spoofed packets would be advocates of TOR. Probably just a troll, luckily providers that host TOR won't care about this type of thing.
It seems like systems shouldn't report abuse (at least automatically) for single packet, no round trip, requests unless its reaching denial of service levels of traffic (and maybe these are). Like in particular for SSH there's no way thats even a valid connection attempt until some sort of handshake has occurred.
This is the IP version of SWATting, patent trolls, framing an innocent person, or using DMCA takedowns to remove the competition. It's basically weaponizing abuse-protection mechanisms to instead attack a target that is disliked. Interesting that the authorities can become a weak link here and be actively weaponized by unscrupulous actors to achieve their aims, but it's not really a new phenomena.
This is likely a very naive question, but how did the spoofer know his IP was participating as an internal Tor node? From what vantage point can that be seen? I imagine internal Tor nodes must know to connect to each other, so it must propagate through Tor. Is the attacker also a Tor node? Is it trivial to map all Tor hosts?
How difficult would it be to highjack this attack by sending these packages to everyone, so that providers like hetzner would get swamped with abuse emails? This way the attack would not work anymore. Either the honeypots would stop sending abuse emails, or the providers would filter those out.
Why not make ISPs responsible for blocking any such traffic. In the end it must originate from someone's network. And really they also should know who their peering partners are and what traffic should be allowed from there.
Internet where you send a packet over the wire and the network takes it and delivers it per RFC. Basically OG Internet. Network of networks of more or less trusted peers.
Or Internet where you need to requisition every connection/circuit be provisined before it is routed, which includes explaining why you need the service, and where any provider in the chain will deny you transit by default? You now must forge an intimate relationship with every middle box between you and the other endpoint. This process must be repeated by everyone on the network. Just operating as a middle box for someone else is now fraught with legal liability; as anything one of your transit's end up doing, you are now considered complicit in.
Both of these architectures of an Internet are equally valid and functional. The society that uses them however is completely different.
I prefer the former, warts and all, and lack of throat to throttle short of the asshat running the software on the other end, over the latter, because with the former at least, we're not creating power nexii to attract asshats to NetOps positions.
With the latter setup, sure, your spam problem has an ostensibly way higher barrier to entry in the form of having to create human trust networks, but the accretion of social power distinctly changes the culture of the net sector, attracting a type of personality that should never, ever be trusted to be given a yay/nay authority over other folks access to a network.
> don't see why verifying that an IP from your own subnet isn't claiming to be from outside it requires everything in your second paragraph.
You're looking at this as a collective update of firewall rules, and content to stop there. I'm more concerned about what that gesture turns into once it's significance percolates out to the public at large. Societies rearrange themselves around technical capabilities. Continue reasoning about how that constraint evolves into new obligations and legal precedents on the network operator, and you should eventually arrive at why I'm content to leave that particular bear unpoked.
There's no in-band solution to this problem, but out-of-band solutions might exist! For example: (1) Notify the destination ISP that you're receiving backscatter. (2) That ISP checks where the packets are coming from, and notifies that ISP. (3) Repeat step 2 until source is found. (4) Quarantine that part of the network until it behaves better.
People are sometimes shocked to learn that the Internet as a whole works because there is a subset of humanity that really, really likes overseeing the most over-the-top pipe game in existence.
Then, as noted in the article, you're trying to prove a negative to someone who doesn't really care at all, which is borderline impossible.
Automated abuse reports of things that are easily spoofed don't justify a report, but might justify a quick check to make sure your box is still operating correctly and hasn't been taken over.
That's the important part.
If they receive another one (or two, or a few) more abuse reports, they assume it is not fixed, and will expect a response then. Which ends up being annoying.
The legitimate answer would include some sort of real-world attestation about you from a trusted third party. Probably the very least, some evidence of your identity and jurisdiction. Maybe including a video call or something. Not just you anonymously claiming you're a good guy over the internet and expecting to be believed.
It's annoying to find someone (or some service) that is willing to attest on your behalf and have that person (or service) be trusted by your provider more than whoever filed the abuse complaint.
>Maybe including a video call or something.
It's annoying to find someone at your provider who will take the time to do this. It's annoying to take my time to have to do this.
My point, overall, was that this is just a really annoying problem.
Nice try Winnie Poo
There has to be some ability to establish baseline trust.
Honest question as someone that is definitely not a networking expert.
In the abstract: if I own the infrastructure and someone uses that infrastructure to hurt someone, that someone who was hurt (or the parties who protect them) are going to come to me asking questions. If I just say "I don't know" and the law doesn't protect my willful ignorance, I'm at best enabling harm; I'm at worst socially or legally liable for negligence.
In the abstract, the systems of human governance recognize harm and seek to mitigate it.
So if I'm peered to a network using me as a bridge to do harm, I can't trust that network when the bad starts to outweigh the good. If I can't establish trust via human methods, I'm gonna cut that network off to protect myself.
(The Internet started as people who had working relationships with each other and grew out from there. Even though the web of connections is much larger and more indirect now, the whole thing is still at its core a human construct and beholden to human standards of conduct, because humans ultimately have their hands on the various plugs that are yankable).
Same with spoofed MAC addresses, email addresses, ARP messages, Neighbor Discovery, MitM TLS certificates ... It's amazing anything works anymore :D
That isn't an argument for not improving things though, just a warning against perfection, if you chase it then you're liable to make really big mistakes that ruin everything.
Now as far as every other mail operator setting up their stuff right such that From spoofing is no longer feasible, well... Can't help ya there. I don't run my email to make money, so the incentive to adopt pathological configs for the sake of maximizing the number of users/Domains who can send from one IP ain't there.
So, I can assure you this is quite common. As a personal note, I know I’m a bit of an exception for operating multiple IP addresses, but I need the flexibility to send packets with any of my source addresses through any of my ISPs. That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
If you actually have your own IP addresses this is normal and expected, but if you're able to use ISP A's IP addresses through ISP B or vice versa that has always been a bug that you are wrong to use.
If you are doing the latter this is firmly in the "reenable spacebar heating" category and I hope your ISPs fix their broken networks.
for those that need more context regarding the "reenable spacebar heating" comment
I worked at a company that hosted some web assets on-prem in one of their branches. They had a 1Gbps connection there. However, at HQ, we had multiple 10G connections and a pretty good data center. So, we moved the web VM to HQ but kept the assigned IP address (a public static from ISP-A). We routed it through a VPN to HQ. The server used our default GW and sent responses with source IP (ISP-A) via ISP-B (10G).
That way, we utilized 10G outbound, even though the inbound was limited to 1G. It was only for GET requests anyway. I know this wasn’t the most optimal setup, and we eventually changed the IP, but it seems like a valid use case.
Scenario 2: We had two connections from two different ISPs (our own ASN, our own /23 addresses). We wanted to load balance some traffic and sent half of our IPs through ISP-A and the other half through ISP-B. It worked fine, but when we tried to mix the balance a bit, we found an interesting glitch. We announced the first /24 to ISP-A and the second /24 to ISP-B, but ISP-A had RP filtering. So, we had to announce all the IPs to them.
The way the RP filter works, as you may guess, means we cannot prepend or anything. All traffic must come through them. If they see a better route for that prefix, they will filter it. For a few months, they refused to fix this, citing security. There’s no shame in security best practices, so I might as well name the ISP—Virgin Media.
Note that the internet with rp_filter is not $20/month. It was more like 5K+/month!! And we did not change it due to lack of alternatives there. But otherwise guess who loses the contract :)
I think even the cheapest 100bucks business plans from many ISPs come with /28 or /29. It is a complete waste because we had like 10 offices with 3-5 persons with laptops and NO servers. The common question from the ISPs is: Do you need some IPs? When we answer no, they give us /29.
"Of course, Agent Bond."
As someone who always enables rp_filter everywhere... I'm very curious why?
I mean, technically those ISPs would be in violation too. You need your own ASN.
Turns out, most multiphomed IPv6 users need provider indepdent addresses, just like with IPv4. And then you need to make sure your all your ISPs allow you to use all your prefixes. On the plus side, it's much more likely to get an IPv6 allocation that's contiguous and that you won't outgrow; so probably you only need one v6 prefix, and you may not need to change it as often as with v4.
Then have ULA addresses for internal network. Those will be routed with tunnels and VPNs. That separates accessing the internet from internal network, and means that don't need to have routable address space.
The only people who would need own address space have data centers and routers.
Except that ULAs don't really work. They are less prioritized than GUAs.
What would it take to get enough network providers to start rejecting traffic from all ASes that don't implement this, so that spoofing was no longer possible?
It's much easier to work on reducing reflection multipliers though, because you can scan (ipv4 anyway) for reflection vectors and yell at people that will respond with 10x the input bytes.
I suppose a difference is that they use unaffiliated parties to send the complaint, instead of contacting the authority directly.
I'm not sure how often this happens in practice but tracing the source of a spoofed packet is possible if you can coordinate with transit providers to follow the hops back to the source. One time JPMorgan worked with Cogent to tell us to stop sending packets with their IP addresses (Cogent is one of the most spoofer friendly tier 1's on the internet btw).
This is the first time I've heard of this being used to target TOR specifically which seems counterintuitive, you would think people sending out spoofed packets would be advocates of TOR. Probably just a troll, luckily providers that host TOR won't care about this type of thing.
> Probably just a troll
Or someone wanting TOR to be treated like nuclear waste, because it offends their surveillance ops.
Unfortunately many people seem to think otherwise and will spaff abuse reports over an errant SYN packet
Internet where you send a packet over the wire and the network takes it and delivers it per RFC. Basically OG Internet. Network of networks of more or less trusted peers.
Or Internet where you need to requisition every connection/circuit be provisined before it is routed, which includes explaining why you need the service, and where any provider in the chain will deny you transit by default? You now must forge an intimate relationship with every middle box between you and the other endpoint. This process must be repeated by everyone on the network. Just operating as a middle box for someone else is now fraught with legal liability; as anything one of your transit's end up doing, you are now considered complicit in.
Both of these architectures of an Internet are equally valid and functional. The society that uses them however is completely different.
I prefer the former, warts and all, and lack of throat to throttle short of the asshat running the software on the other end, over the latter, because with the former at least, we're not creating power nexii to attract asshats to NetOps positions.
With the latter setup, sure, your spam problem has an ostensibly way higher barrier to entry in the form of having to create human trust networks, but the accretion of social power distinctly changes the culture of the net sector, attracting a type of personality that should never, ever be trusted to be given a yay/nay authority over other folks access to a network.
I don't see why verifying that an IP from your own subnet isn't claiming to be from outside it requires everything in your second paragraph.
You're looking at this as a collective update of firewall rules, and content to stop there. I'm more concerned about what that gesture turns into once it's significance percolates out to the public at large. Societies rearrange themselves around technical capabilities. Continue reasoning about how that constraint evolves into new obligations and legal precedents on the network operator, and you should eventually arrive at why I'm content to leave that particular bear unpoked.
It never stops at the technical. Ever.
Just acquire a few boxes that don’t block spoofing outbound SYN packets and start spamming random IP’s from random IP’s with SYN packets.
It will generate a shitload of abuse emails and accomplish mostly nothing except fill up disk space with useless emails and such.
At the end of the day, the internet is people.