> During device initialization, if the system identifies itself as a user terminal, the initialization script automatically writes 41 SSH public keys into /root/.ssh/authorized_keys. Notably, port 22 on the UTA remains open to the local network at all times.
Forty-one? So who does not have root access to "your" user terminal?
On a more serious note, is this any different from ISPs having a remote management system for ISP provided routers? In terms of privacy, if SpaceX didn't have access to the user terminal, they could still just capture your traffic on the sattelite or the ground stations
On a more serious note, is this any different from ISPs having a remote management system for ISP provided routers?
Maybe, but in more and more European countries, ISPs are required to accommodate you hooking up your own router/modem. E.g., I am on fiber and if I want to I can hook up my own router directly to fiber with an SFP+ module (I currently use the ISP-provided media converter, but my own router). Lots of tech users use their own Ubiquiti/OPNsense/OpenWrt routers, so no remote management.
I wonder if this requirement applies to Starlink as well, since they are an ISP.
As far as I'm aware, they are only required to allow you to use your own router.
DSL tech is far simpler and it's always a combo unit so I could see a case where you would be allowed to bring your own DSL modem.
But it just doesn't work like that for DOCSIS or GPON where the cable modems or ONTs these days do much more than just media conversion - SIP, PPPoE, IGMP, etc. even if they don't do Wi-Fi (so ISPs don't call them "routers" - except SingTel, which uses "ONR" to distinguish these units because they are in fact routers for IPTV and SIP).
For all of those modems/ONTs, the firmware updates and the configuration for telephony/SIP and PPPoE are controlled by the ISP and also tested to work with their OLT or CMTS so it's just not possible for the ISP to guarantee support for any random modem or ONT.
And to support the advanced configuration required these days for VoIP, IPTV, etc. on the "modem" or "ONT", ISPs basically have a backdoor called TR-069 which is really not too dissimilar to what Starlink has access to with their SSH keys.
Even if you get "true" dumb modems or ONTs which do not do any routing whatsoever, the device on the other side still has full control over your dumb device via the DOCSIS provisioning process or GPON's OMCI. Starlink seems to be using SSH instead of building a whole protocol - because satellite tech is proprietary and doesn't need to work on other hardware.
So, I find that it's highly unlikely that the ISP is officially required to support a user supplied modem, although I haven't consulted the EU laws on this.
At most, I think using your own router would require the EU ISPs to provide bridge mode support, but that's not special to EU. However, the TR-069 backdoor is still active even with bridge mode.
It can be fairly easy to stop TR-069 with a "dumb" ONT (usually SFP) but ISPs can and will notice that. Whether they allow it is up to them.
>DSL tech is far simpler and it's always a combo unit so I could see a case where you would be allowed to bring your own DSL modem.
Not really, when you want to increase the bandwith, e.g. with vectoring[1], you need to have all neighbor modems to participate, which prevent free modem choice for the users.
>But it just doesn't work like that for DOCSIS or GPON where the cable modems or "ONT" router combo units these days do much more than just media conversion - SIP, PPPoE, IGMP, etc.
In Belgium, the ONT is just media conversion these days, SIP is done on the provider box, so you can have your own GPON SFP.
>so it's just not possible for the ISP to guarantee support for any random modem or ONT.
The ISP doesn't have to guarantee support to let you use your own hardware. It just have to give you the specs to use it and let you plug the ISP box if you can't configure vlan of dhcpv6 client.
Each country regulator in the European Union have to set it's own regulation, but the BEREC (Association of UE telecom regulators) guidelines say that in most case, the free choice of router and modem is what's is required by the EU decisions https://www.berec.europa.eu/sites/default/files/files/docume...
> But it just doesn't work like that for DOCSIS or GPON where the cable modems or ONTs these days do much more than just media conversion - SIP, PPPoE, IGMP, etc. even if they don't do Wi-Fi (so ISPs don't call them "routers" - except SingTel, which uses "ONR" to distinguish these units because they are in fact routers for IPTV and SIP).
At least in Finland the norm is that you can use your own DOCSIS modem from any manufacturer, you just tell the ISP your modem's MAC address.
> So, I find that it's highly unlikely that the ISP is officially required to support a user supplied modem, although I haven't consulted the EU laws on this.
This. I can just provision in the backdoor interface on the modem with a config file anyways and gain access.
Plus depending on model (like Arris modems), I can do things like set the password of the day seed (away from the factory default) to further lock it down and gain management access remotely.
Not the case in the U.K., dal or fibre doesn’t matter, domestically you typically have a standard sfp or other converter from anywhere which presents it as Ethernet which you then run pppoe over.
Tr069 allows the isp to remotely configure their equipment which most people are happy with, but if you want to use your own then that’s fine, and obviously unless you enable it the isp won’t configure your router or any other equipment.
Provider will not support my modem that is for sure so if you have any issues you are on your own. I use my own Mikrotik + Zyxel PMG3000 GPON SPF and no issues at all.
If a normal ISP wants to operate in country a, they need infrastructure in country a. This means they either follow country a's laws or that infrastructure gets seized.
Starlink could just as well be operating entirely from the US, and there's very little that foreign governments could do to stop them if they break some foreign laws. They could make payments and shipping complicated, which is probably why Starlink would rather comply if the requests are somewhat reasonable, but Musk has indicated multiple times that he's willing to stand up to unreasonable restrictions if the need is dire enough.
This is not at all how laws actually work. If you sell a product to customers in a specific country, you generally have to comply with local laws. You might be able to avoid this if you're very small and your service is entirely virtual. However, as long as Starlink needs to provide their customers with physical hardware, there are numerous ways to enforce regulations.
And you can always go after people - and I mean both Starlink executives and customers.
Some countries require you to have ground infrastructure in the country to operate satellite systems. Starlink's architecture also means they need to have a lot of ground stations all over the place. They can skip some areas if they need to and still offer service, but they'd seriously struggle to provide a global service by using only US ground stations.
> I am on fiber and if I want to I can hook up my own router directly to fiber with an SFP+ module
I don't think you quite understand how this works.
The ISP controls whatever the other end of that fiber is plugged into. It doesn't matter if the medium is fiber, or copper, or a piece of string. The ISP always has control of the other side of the customer interface. It doesn't matter if the box physically resides in your home or not.
In the case of Starlink, it's all contained within one box.
In the case of DOCSIS (cable), you may physically own the modem, but the ISP controls the firmware it netboots and has full remote admin to the device.
It gives them access to the LAN so they can, for instance, figure out how many internet gadgets your house has and sell that information to advertisers, or do even worse than than.
AFAIK, in America, ISPs are required to permit user-provided modems as long as those modems are technologically compatible. I believe the Television Viewer Protection Act of 2019 did this, although it was already the norm for ISPs to permit third party modems before this.. I guess because they knew if they pushed the issue they'd lose anyway, given the precedent of telephones, cable cards, etc.
The problem then with Starlink is nobody is manufacturing compatible third party Starlink terminals, at least yet.
Note that such capture would be quite terrible for performance, not only requiring disabling any hardware offload (a great router might be able to route a few hundred megabit in large packets without offload assuming it doesn't do anything else) to make packets visible for capture, but it would also have to stream the output back to the adversary over the uplink as you would be limited to at most a few gigabytes of local, extremely slow storage, giving no means for local offline analysis...
The risk of access to the router is more that they can access your network and touch unprotected and vulnerable things rather than active monitoring.
And most of that wouldn’t be useful. We use encryption for almost everything now for a reason.
No, wiretaps on modern networks do not rely on backdoors, or even big labeled front doors like SSH, on individual subscriber devices. Instead it is built into the lower level routing. When an ISP gets a warrant (or whatever relevant document your country uses) they configure their routers to tag all of your traffic and mirror it to a server to be recorded. It’s entirely invisible to the subscriber, and highly automated.
I wonder who would be best equipped to see if any of those keys are traceable to individuals involved in special government affairs lately? There have been some good leaks...
could simply be 41 instances of the same server in 41 regions, not necessarily a cause for concern. Starlink is a global service after all. I'd be more concerned if 41 instances were sharing one key.
> I'd be more concerned if 41 instances were sharing one key.
Dozens/hundreds/thousands of web servers servers can easily share one private key in a certificate, public keys offer even more options on sane designs. Directly authenticating 41 servers using ssh-keys is just poor, slap dash engineering.
I would argue reusing private keys worldwide is slapdash engineering. You generally want to minimize exposure in the event of a breach, not maximize it.
> I would argue reusing private keys worldwide is slapdash engineering
I wasn't suggesting it, and frankly can't see how that could be a solution in this instance. I was making a comparison against current practices on a harder problem to solve , i.e. safely scaling a single private key in an SSL certificate across many servers is solved today without a 1:1 server to certification ratio
It is not, amd I can't see how my earlier comment can be read as recommending that. This is a solved problem for private keys (using load balancers, for example) , so public keys are lower-hanging fruit than that.
Edit: upon rereading, I cam see how the word "share" would be ambiguous in the context of if a private key. I meant "jointly make use of", rather than "distribute copies throughout the fleet". I have exited my root comment to make my meaning clearer.
A better idea would be the terminal trusting one or two core certificate authorities and then those authorities creating time limited certificates when needed.
So the terminal accepts "sshauthority1"
Then the 41 remote sites contact sshauthority1 to get a 1 hour (10 minutes, 10 days, whatever) long certificate for "site18"
If a remote site is compromised sshauthority1 no longer issues certificates, and within an hour (10 minutes, 10 days, etc) the remote site can no longer reach the terminals.
Revoking a key from that many terminals (many of which will be offline) if one of the 41 keys is exposed is not trivial.
Now if sshauthority1 is compromised then you've got the same issue with rotation (although can CRL it), but it's easier to secure one or two authorities than 41 keys.
Is that normal? I would imagine that if I were managing such a large deployment, I would just use a CA for the keys and then issue CA signed private keys so that I don't need to add a bunch of random ones to authorized_keys
I’m a single user and my authorized_keys is 25 lines. I have different yubikeys in laptops, keys on iPads and iPhones, and secure enclave keys on macs.
I imagine starlink has more than 1-2 sysadmins. I think a hundred pubkeys would be reasonable.
It seems wrong that each individual sysadmin human in Space X would need to (a) login to my device remotely, and (b) require individual credentials to do so.
Having some way to remotely push updates, and having some kinda of (preferably with your consent!) remote access might be reasonable, but I would expect that to be via some kind of intermediate gateway/app/something and not direct from a sysadmin’s individual account.
> Android Emulator is downstream from the QEMU emulator. It adds support for booting Android devices, emulates typical Android hardware (OpenGL, GPS, GSM, Sensors) and a GUI interface. The android emulator extends qemu in various ways.
> Drawing on existing research [3], our preliminary analysis of these programs and configurations suggests that the network stack architecture is somewhat similar to DPDK [4], mainly relying on a user-space C++ program to bypass the kernel for handling network packets.
The way it usually works is that the initial packets are handled in software but once the endpoints are established it flows through hardware. Sometimes certain patterns are always handled in software. The software could be a patched kernel or a XDP style kernel bypass.
Source: worked peripherally on an Intel Puma cable modem router/gateway that used DPDK or something like it. So I'm not 100% sure, but it is an educated guess.
Why would it be any less efficient than processing the packets in the kernel? There's a way to map the hardware queues into userspace (the article talks about the system being DPDK-like). At that point why does it matter that the polling code isn't in the kernel?
Most hardware >100Mbps has hardware offload - ie. the hardware is told which packets to send where, and software doesn't touch individual packets (except rare packets like ping).
In networking, it is the norm to measure performance in packets per second, so with small packets. Unless you're performing DPI or encryption, routers only use the headers to take routing decisions, so whether the payload is 10 bytes or 1000 bytes does not matter: the processing cost will be identical. Only the hardware bandwidth will matter for large packets, though this is rarely the issue (I've hit DDR4 limits once using XDP, and fixed by adding another stick of memory);
Forty-one? So who does not have root access to "your" user terminal?
On a more serious note, is this any different from ISPs having a remote management system for ISP provided routers? In terms of privacy, if SpaceX didn't have access to the user terminal, they could still just capture your traffic on the sattelite or the ground stations
Maybe, but in more and more European countries, ISPs are required to accommodate you hooking up your own router/modem. E.g., I am on fiber and if I want to I can hook up my own router directly to fiber with an SFP+ module (I currently use the ISP-provided media converter, but my own router). Lots of tech users use their own Ubiquiti/OPNsense/OpenWrt routers, so no remote management.
I wonder if this requirement applies to Starlink as well, since they are an ISP.
DSL tech is far simpler and it's always a combo unit so I could see a case where you would be allowed to bring your own DSL modem.
But it just doesn't work like that for DOCSIS or GPON where the cable modems or ONTs these days do much more than just media conversion - SIP, PPPoE, IGMP, etc. even if they don't do Wi-Fi (so ISPs don't call them "routers" - except SingTel, which uses "ONR" to distinguish these units because they are in fact routers for IPTV and SIP).
For all of those modems/ONTs, the firmware updates and the configuration for telephony/SIP and PPPoE are controlled by the ISP and also tested to work with their OLT or CMTS so it's just not possible for the ISP to guarantee support for any random modem or ONT.
And to support the advanced configuration required these days for VoIP, IPTV, etc. on the "modem" or "ONT", ISPs basically have a backdoor called TR-069 which is really not too dissimilar to what Starlink has access to with their SSH keys.
Even if you get "true" dumb modems or ONTs which do not do any routing whatsoever, the device on the other side still has full control over your dumb device via the DOCSIS provisioning process or GPON's OMCI. Starlink seems to be using SSH instead of building a whole protocol - because satellite tech is proprietary and doesn't need to work on other hardware.
So, I find that it's highly unlikely that the ISP is officially required to support a user supplied modem, although I haven't consulted the EU laws on this.
At most, I think using your own router would require the EU ISPs to provide bridge mode support, but that's not special to EU. However, the TR-069 backdoor is still active even with bridge mode.
It can be fairly easy to stop TR-069 with a "dumb" ONT (usually SFP) but ISPs can and will notice that. Whether they allow it is up to them.
Not really, when you want to increase the bandwith, e.g. with vectoring[1], you need to have all neighbor modems to participate, which prevent free modem choice for the users.
>But it just doesn't work like that for DOCSIS or GPON where the cable modems or "ONT" router combo units these days do much more than just media conversion - SIP, PPPoE, IGMP, etc.
In Belgium, the ONT is just media conversion these days, SIP is done on the provider box, so you can have your own GPON SFP.
>so it's just not possible for the ISP to guarantee support for any random modem or ONT.
The ISP doesn't have to guarantee support to let you use your own hardware. It just have to give you the specs to use it and let you plug the ISP box if you can't configure vlan of dhcpv6 client.
[1]: https://en.wikipedia.org/wiki/VDSL#VDSL2_vectoring
At least in Finland the norm is that you can use your own DOCSIS modem from any manufacturer, you just tell the ISP your modem's MAC address.
Not for GPON, though.
Ziggo (called UPC in other EU countries) uses DOCSIS. The instructions on how to use your own DOCSIS modem are at the following link (in Dutch): https://www.ziggo.nl/klantenservice/apparaten/wifi-modems/ei...
Edit: it really is using your own modem. It's not about putting it in bridge mode.
Plus depending on model (like Arris modems), I can do things like set the password of the day seed (away from the factory default) to further lock it down and gain management access remotely.
Tr069 allows the isp to remotely configure their equipment which most people are happy with, but if you want to use your own then that’s fine, and obviously unless you enable it the isp won’t configure your router or any other equipment.
Starlink acts far more than a media converter.
If a normal ISP wants to operate in country a, they need infrastructure in country a. This means they either follow country a's laws or that infrastructure gets seized.
Starlink could just as well be operating entirely from the US, and there's very little that foreign governments could do to stop them if they break some foreign laws. They could make payments and shipping complicated, which is probably why Starlink would rather comply if the requests are somewhat reasonable, but Musk has indicated multiple times that he's willing to stand up to unreasonable restrictions if the need is dire enough.
And you can always go after people - and I mean both Starlink executives and customers.
eg they could outright ban the sale of StarLink products, ya know, being in charge of the laws and all
I don't think you quite understand how this works.
The ISP controls whatever the other end of that fiber is plugged into. It doesn't matter if the medium is fiber, or copper, or a piece of string. The ISP always has control of the other side of the customer interface. It doesn't matter if the box physically resides in your home or not.
In the case of Starlink, it's all contained within one box.
In the case of DOCSIS (cable), you may physically own the modem, but the ISP controls the firmware it netboots and has full remote admin to the device.
The problem then with Starlink is nobody is manufacturing compatible third party Starlink terminals, at least yet.
The risk of access to the router is more that they can access your network and touch unprotected and vulnerable things rather than active monitoring.
No, wiretaps on modern networks do not rely on backdoors, or even big labeled front doors like SSH, on individual subscriber devices. Instead it is built into the lower level routing. When an ISP gets a warrant (or whatever relevant document your country uses) they configure their routers to tag all of your traffic and mirror it to a server to be recorded. It’s entirely invisible to the subscriber, and highly automated.
Dozens/hundreds/thousands of web servers servers can easily share one private key in a certificate, public keys offer even more options on sane designs. Directly authenticating 41 servers using ssh-keys is just poor, slap dash engineering.
I wasn't suggesting it, and frankly can't see how that could be a solution in this instance. I was making a comparison against current practices on a harder problem to solve , i.e. safely scaling a single private key in an SSL certificate across many servers is solved today without a 1:1 server to certification ratio
It is not, amd I can't see how my earlier comment can be read as recommending that. This is a solved problem for private keys (using load balancers, for example) , so public keys are lower-hanging fruit than that.
Edit: upon rereading, I cam see how the word "share" would be ambiguous in the context of if a private key. I meant "jointly make use of", rather than "distribute copies throughout the fleet". I have exited my root comment to make my meaning clearer.
So the terminal accepts "sshauthority1"
Then the 41 remote sites contact sshauthority1 to get a 1 hour (10 minutes, 10 days, whatever) long certificate for "site18"
If a remote site is compromised sshauthority1 no longer issues certificates, and within an hour (10 minutes, 10 days, etc) the remote site can no longer reach the terminals.
Revoking a key from that many terminals (many of which will be offline) if one of the 41 keys is exposed is not trivial.
Now if sshauthority1 is compromised then you've got the same issue with rotation (although can CRL it), but it's easier to secure one or two authorities than 41 keys.
Perhaps Elon _doesn't_ have a brain the size of a planet after all.
I imagine starlink has more than 1-2 sysadmins. I think a hundred pubkeys would be reasonable.
Having some way to remotely push updates, and having some kinda of (preferably with your consent!) remote access might be reasonable, but I would expect that to be via some kind of intermediate gateway/app/something and not direct from a sysadmin’s individual account.
Teardown of the SpaceX Starlink User Terminal https://news.ycombinator.com/item?id=25277171 (December 2, 2020 — 158 points, 138 comments)
Anyone has links to resources about how to emulate a firmware that connects to external devices (GPS here), any ready solutions?
> Android Emulator is downstream from the QEMU emulator. It adds support for booting Android devices, emulates typical Android hardware (OpenGL, GPS, GSM, Sensors) and a GUI interface. The android emulator extends qemu in various ways.
If one is doing 1Gbps of traffic which is 100 byte UDP packets, that's a million packets per second you're gonna need to process.
A 1Ghz CPU only then gets 1000 cycles to process each one...
Very doable, but certainly not easy unless your engineers like hand coding assembly and having to think about every lookup table trick in the book...
The way it usually works is that the initial packets are handled in software but once the endpoints are established it flows through hardware. Sometimes certain patterns are always handled in software. The software could be a patched kernel or a XDP style kernel bypass.
Source: worked peripherally on an Intel Puma cable modem router/gateway that used DPDK or something like it. So I'm not 100% sure, but it is an educated guess.
Specifically for cases of forwarding DPDK-style approach can be faster because it will incur fewer buffer copies.
Starlink only does 25-200Mbps and average packets are like 7-8x larger so at most you're doing ~36000 PPS which is pretty manageable even on 1Ghz
100 byte?? Starlink has regular 1500 byte MTU.