This is a WIP. Last update 2024-09-21.
I wrote a fairly bland history of is.xivup.com, but I wanted to dive into some of the technical aspects of the site that are interesting (aka: this post). The code is not open-source right now due to “reasons”, so you get this blog post instead.
Where it is now
Running scc on the codebase yields the following:
───────────────────────────────────────────────────────────────────────────────
Language Files Lines Blanks Comments Code Complexity
───────────────────────────────────────────────────────────────────────────────
Go 35 5139 594 235 4310 721
Plain Text 22 22 0 0 22 0
Go Template 14 784 32 5 747 24
TypeScript 10 1662 194 157 1311 173
JSON 7 141 2 0 139 0
Sass 3 387 89 76 222 0
LESS 2 483 64 63 356 0
Markdown 2 181 21 0 160 0
YAML 2 583 22 27 534 0
HTML 1 68 2 0 66 0
JavaScript 1 2 0 1 1 16
Shell 1 24 8 3 13 1
Systemd 1 15 2 0 13 0
TypeScript Typings 1 68 10 0 58 0
───────────────────────────────────────────────────────────────────────────────
Total 102 9559 1040 567 7952 935
───────────────────────────────────────────────────────────────────────────────
The backend is entirely in Go, and the front is a mix of various tech. There are some interesting bits in all these holes, so lets dive in!
Pinging Hosts
Across all the possible FFXIV servers, there are 133 IP addresses hosting game servers (as of this writing), plus an addition 12 lobby servers. I ping each of these servers 3 times once a minute. While I could use existing CLI tools, that’s not a way to learn, so I wrote my own ping loop.
I open an icmp.ListenPacket, which allows me to keep it open and just fire off a bunch of pings at once. I have a for-loop that sends all 145 pings at once. I then wait for the responses in a loop, recording each as they come in. You need to be careful that you tag the packets with data that when echo’ed back, you know you were the one that sent them.
One thing I was worried about here was latency of sending the packets in a loop. I timed how long it takes to send all 145 pings, and I ran into something unexpected. I wrote/tested this code (in 2017) on Windows, and was seeing that it was taking me 30-50ms to send 145 pings, which seemed crazy to me. Now adays, Windows takes between 3ms-20ms (avg of 6ms) to send all 145 pings, which is better. Linux has always been much faster for me, where it can send all the pings in 0.5ms-4ms (avg of 2ms). Looking into it now, I likely was tripping over a timer resolution issue, where Windows 10 prior to April 2018 has a 16ms resolution (compared to 0.5ms resolution now adays).
This code has been working pretty solidly since I wrote in it 2017. There have been minor tweaks, but it’s just worked.
Portscanning
This is meant to check that the game servers are up and accepting connections. This doesn’t tell us if the game is available to play on or not, it’s only there to detect any connection issues.
Finding the ports to scan
SquareEnix helpfully tells us what ports their game operates on, which gave us a good starting point. Using Wireshark, plus spending a bunch of time creating a character on every server in every DC (I only did this once), I was able to get some heuristics about the ports and IPs they use. Things I learned:
- The IP/port for a given realm changes with each server restart or game patch. (this was true a few years ago, and it may be even more random now adays with cross-realm visiting)
- The ports they ever opened up was limited. Game servers always use the ports in the range of 54992, 54993, 55006, 55007, 55021 through 55028.
- Lobby servers only ever use port 54994
- Lobby servers have DNS records, game servers are not looked up by DNS (instead the lobby server reports the IP/port to use for a given game server).
- Game servers for a given “logical data center” are always clumped together and occasionally move around some slightly within the IP blocks for that region.
nmap is my friend here for keeping this list up to date. I have though about automating updating the list of IPs for game servers, it only happens every few months, so it’s not worth the time that it would take to build out properly. Balancing time/benefit here is important for a side project. It would be nice to automate this, but I could spend that time gathering other data that would actually be useful all the time.
Scan those ports!
There are a lot of ports to scan (133 game servers * 12 ports + 12 lobbies * 1 port = 1608 ports). Scanning a port is rather simple, I’m just opening a TCP connect to each one (using net.Dialer DialContext), try reading a few bytes, then closing the socket. Errors at various steps in here help tell us the state of the game or lobby servers. If the connect fails to be established or their servers close the connection, then I know the servers are likely offline. But if I see no errors opening or reading from the socket, then I deem the servers online.
I spin up a go-routine for each port scan, and it seems to work pretty well. As latency isn’t super important here, as long as their game servers don’t filter the data, and my server can send the connect requests quickly, then all behaves fine (I also don’t want to spend all that time sending the requests sequentially). Opening each connection can also add to the connection count of a firewall (which I talked about here), so I tripped over a few things while building this.
This also has been working pretty flawlessly since I wrote it.
TODO
This post is a work-in-progress. Other topics from this I’ll write about still:
- Caching results
- Dropping Docker for direct hosted
- Move from Nginx to Caddy
- Reducing logging
- Move from self-managed cert to Nginx/Caddy
- Tuning SQLite
- Picking a SQLite driver
- Passing data between scanners and main