This detective story is absolutely entertaining and is devoted to all sysadmins, DevOps engineers, and related specialists. Any resemblance to actual persons is purely coincidental. Based on real events. Created by our CTO. Enjoy!
There are days you’re happy just to make it through. And there are Friday evenings you wish you’d rather not have made it. After the liters of coffee and endless lines of shell commands, I’d sell my soul for a good cigarette — any cigarette, for that matter. It’s been five days since I quit smoking — four days and fourteen hours, to be precise — and it’s been hard. No whining, I thought. Just find something to focus on.
The work helped. It’s been my safe haven these days, so much I started dreading going back home. I got other things to do, sure — but none of them mattered if I couldn’t get into the flow state. A good challenge became my escape, and it was time for the next round of tasteless coffee to help me scroll through the list of boring issues.
I took the mug from the coffee maker stand and watched the steam rise from the black surface. In the dim lights of the office kitchen, it looked like thin smoke. I sighed and walked back to the SRE room. Hang on just a little longer, it’ll get easier…
She was sitting on the edge of my desk, high heels a few inches in the air, an evening dress the colour of dark wine, and a long umbrella folded on her lap. She wore no jewellery — instead, a smile to die for. Which turned into a smirk as soon as she saw me:
“Rough day?”
“A week. Hunting season?” — I nodded at her dress. Nancy narrowed her eyes, processing that, then her smirk turned into a satisfied grin:
“Just a date I’m late for. Before I leave, I’ve got a task and I want to make sure you don’t miss it. I bet you’d like to leave this one unnoticed.”
“No more bets for him,” I heard Nick chuckle from his corner desk, and I tried hiding my frustration in a sip of coffee.
“Lost another one?”
“Actual breathable air around him, can you believe that? A no-smoking month!”
Nancy burst out a wicked laughter:
“As much as I’d like to hear the full story, I’m in a hurry. Look, I assigned an issue to you — it’s about helping Josh with hidden IPs.”
Hidden IPs, I thought. Find me a manager that can be any vaguer than Nancy, and I’ll quit smoking for good. Well, at least it’s not an emergency.
“And make it a priority, please. The guy’s been trying to get some assistance from your team the whole week, and the client is getting anxious about this location thing. And apparently, none of you could figure this out.”
I gritted my teeth.
“None? Has Henry looked at it?”
She tossed her hair in a gesture of defiance, then stood up:
“He did, and he couldn’t help Josh.”
“Well, then nothing can help Josh,” I snarled.
She straightened her dress for a second, then walked to the door.
“Listen, do whatever you need to do, just get it done. It’s all in the task. Talk to the guy,” she said back through the narrow slit of the closing doorway.
I took a deep breath, trying to stay calm. The air full of Nancy’s perfume made me regret this immediately, and I coughed a couple of times.
“Do you know what she’s talking about?” — I looked at my teammate. Nick took off his headphones:
“If it’s #2907, I might. Josh asked the other day if one can find out the real IP address of a user if they are behind some kind of VPN. I explained possible ways, proxies, real_ip_header in Nginx and so on. He didn’t ask for help and seemed only interested in the theory.”
“There must be more to it. Why else would she act like this is a big deal?”
“Nancy,” he shrugged. “Maybe Josh doesn’t understand something but doesn’t want to admit it, so he’s making the problem seem bigger than it is.”
Indeed, Josh wasn’t the brightest of the client’s developers — and arrogant, to boot — but a job is a job. He could complain to a manager that the SRE team didn’t help him — but it looks like there was nothing to help him with.
Something was off.
I decided to check #2907, not really hoping to find anything useful there. “It’s all in the task” normally meant there would be no description. Find real IPs of visitors hidden with VPN, the title said. The description was empty but the issue was a blocker for another one: Display visitor’s country in the admin panel, assigned to Josh. Some time logged, no comments left.
But #2907 had a comment from Henry: Couldn’t find invalid IPs in the web server log, can’t reproduce.
“Dude?” Nick was standing in front of my desk with his backpack over one shoulder. He’s about to leave, meaning I only have two hours before Eric starts his graveyard shift.
“Yeah, bye. Anything to share?”
“All is closed, left my notes in the chat. See you on Monday” — Nick left into the bleak drizzle, turning the room lights off as he went.
If I don’t crack it in two hours, #2907 will become Eric’s problem. We had an unspoken Code of SRE, the rule of not leaving the job undone for a teammate to pick up — unless you can’t deal with it on time. I knew I didn’t have enough information then — but this is how every investigation begins.
I needed a case file, so to speak. Chat archives of daily requests, a bottomless pit of technological crimes committed by shady developers and left for us to struggle with, should have everything I need. I shivered at the thought of reading through these texts — but Henry’s on vacation and calling him must only be a last resort, so I started filtering. The sound of keystrokes in the room mixed with the steady rhythm of the rain outside; no coffee left in the mug, but I ignored it.
A minute later, I found Nick’s dialogue with Josh. Just like he said, wanted to know if IPs like 10.x.x.x can be traced to their real users. These are local network IPs… Web server could have only registered them if it’s behind a reverse proxy and not configured to use real_ip_header, I thought. And that is exactly what Nick told him. Then he checked the server, and found it’d been “configured properly.” But Josh then said he got what he needed for his task.
Sloppy work. He could’ve checked if what Josh was saying about IPs was even true. Trusting a developer… A mistake you don’t make twice. I kept looking for another chat, the one Josh must have had with Henry afterwards. This one had Josh asking to trace what country 10.93.0.267 was from, and Henry being Henry meticulously explaining how 10.93.0.267 had an absolutely impossible value and just won’t work. He remained polite up until the very end of the conversation when Josh recommended him an article to prove that “such IP is normal because it has four digits and belongs to class A range.” Somehow Henry’s point that an octet can not be greater than 255 didn’t convince the developer. “You could’ve at least been a professional and admitted you didn’t know the answer.” Henry suggested he check if the IP was even reachable, then asked where he got it from, but Josh never got back.
Angered, I squinted at my empty mug and got up to refill it. Jerk. I pondered on collecting the chat logs and dumping them into the issue for Nancy to see but realized it’d be just a waste of time. The time I’d rather spend on actually solving the problem.
I felt a headache slowly creeping in from the back of my skull. I shut my eyes tight, pressing fingertips on the eyelids and trying to piece things together through the whirring sound of the coffee maker and drumming of the downpour. So Josh is tasked with detecting visitors’ countries by their IPs, he’s got some weird addresses and thinks it’s because the users are using some public VPN. But the project doesn’t have its own VPN, and a public VPN would’ve definitely shown an external IP. And the IPs themselves weren’t just wrong — they were impossible. No wonder Henry couldn’t find them in the logs. Which raised a good question: how did Josh even come up with these addresses in the first place?
I brought the mug back to my laptop and sent a simple “hi Josh” — with the timezone difference between us, it was almost noon for him. But I’m not delusional: time is a luxury, and I didn’t expect a prompt reply. I had some stuff to check before I got any, so I connected to the server and started digging. First, confirmed what I already knew from Nick and Henry: the reverse proxy was configured to pass real IPs to the website and the access logs had no local addresses in them. It was a dead end, but the one I expected to reach.
I inspected the website a little and looked for other log files, just to find none. Drupal, PHP 7.4, a bunch of custom modules, nothing out of the ordinary. I entered the database shell and scanned through the list of tables, but nothing caught my eye. Another dead end.
I needed a clue, however minuscule, anything that would help me find the IPs Josh kept asking about. I frantically typed a grep command to look for the invalid IP recursively starting at OS root. I was that desperate.
Was there anything I didn’t know about the setup? Likely, but only the website part; infrastructure and OS-wise, I knew these like the back of my hand, a generic single server hosting. Annoyed with not having any progress, I aborted grep execution and went back to MySQL shell: if by some chance Drupal was collecting visitor’s data, it all would end up there. A few sips of cold coffee later, I found what seemed like a presentation of access logs in a table called table4. I started typing a SELECT query to check for the damned IPs when the Slack icon flashed with a message. Josh.
After briefly exchanging formal greetings, I cut straight to the chase:
“Where have you found 10.93.0.267?”
I seemed to have lost him there, so I went back to the shell to run my query. The IP actually was there once, two days ago. I crafted another query to check for what would be considered proof of my insanity by any of my teammates: more “impossible” IPs. There were all kinds of invalid addresses: 0.991.0.118, 14.1.1.003, to name a few. I couldn’t have missed those in the weblogs.
“In the log”, came Josh’s response. The most frustrating thing about it was that he wasn’t lying, we just had different meanings for this word. I asked him for a screenshot and got a picture of the Drupal admin page showing the same data I stared at in my terminal.
“The data in the log is incorrect”, I typed. “How does it appear there?”
“A visitor opens the page, we store their IP in the log — there’s a plugin for that. What’s incorrect about it?”
I waved my fingertips in the air before touching the keyboard.
“There can’t be such IPs in the world,” I said. “Try pinging it.”
“I did. It’s unreachable.”
That made me pause and reconsider my entire career for a moment. I’m pretty confident about my networking, and unreachable means inaccessible. It exists, it’s somewhere out there, but there’s no path to it for your computer. I could only find one explanation for this: if his ping command tried parsing the IP address as a domain name. Unlikely though… The error would still be different, anything but “unreachable”.
Before I asked him for proof, Josh had already sent me another screenshot.
ping: invalid address
“See? Unreachable”, the caption said.
I looked out the window at the street lights, weighing my options. A familiar bubble of existential crisis swelled up inside me. For the first time since I started investigating this issue, I felt the tingle in my throat, urging me to get a smoke and be done with it. I could quit this job instead and do something actually rewarding. I could be a farmer, have permanently tanned skin and spend my days surrounded by chicks. I’d name a rooster Josh, the only reminder of a life I left behind.
Dream on.
I finished my coffee in a single gulp and left Josh’s message unanswered, switched back to the terminal and looked up the timestamps from the Drupal table in the access log. Now that I was sure it was just a matter of presenting the data in the wrong way, I only needed to find what it was initially that the website somehow converted into non-existent IPs. The column that normally held an IP address had 0991:0db1:18a3:0000:0000:8a2e:0370:7334 instead. It dawned on me then.
“Josh, is that plugin custom?”
“It had been there before I joined, developed internally. Why?”
“Whoever coded it, was not aware of IPv6,” I paused to take a screenshot of the log I filtered for the timestamps of invalid IPs. “This is what your log entries look like from the web server perspective. It’s not 0.991.0.118, it’s IPv6 0991:0db1:18a3:0000:0000:8a2e:0370:7334 — and the plugin does its best to convert it to a valid IPv4 by cutting off everything that in the author’s opinion doesn’t belong to an IP.”
I had little hope that Josh would go Googling what IPv6 is, but he didn’t respond to my explanation. A sudden thought raced through my head, and I went to the project’s git repository just for curiosity’s sake. I heard quick footsteps for a few seconds before Eric entered the room.
“What’s up?”
“Six years, Eric,” I said absently. “And nobody even cared to open this log until the client wanted to see the countries.”
He opened the window to let some chill air in, then turned around and threw a concerned look at me. Street lamps outside left quivering stripes of light on the office floor and my chair. Except for the laptop screen glow, the rest of the room was covered in murky shadows.
“Can you get any more noir?” — he asked cautiously.
“Perhaps… If I had a detective story to tell.”
