Bertrand Florat
Tech articles
Others articles
Projects
Cours
Contact
The Biggest WTFs of My Career
Bertrand Florat - Aug 23, 2026
Introduction
There is a good way to check if you work in a reasonably structured, competent and serious work environment: count the 'WTF' you say in a day. Up to two or three, it's OK. More than ten, run!
During my career, I worked on many IT projects in different contexts and I often observed basic errors, misunderstandings and a lack of tests, but some of the issues I had to deal with are worth reporting. Some are strange, others inexplicable, sometimes even poetic, meaningful, or romantic. I learned many things while dealing with them.
Most of these issues were made by intelligent people corrupted by the segmentation of work (Taylorism doesn't work in IT, the waterfall process doesn't work). Humans have a great propensity to disconnect their judgement in salaried work, and when the work is supposed to have been done previously by other departments.
Here is my personal selection, grouped by family — because after a while, the same patterns keep coming back.
Contents
- The Credit Card and the UUID
- The Ballooning Problem
- The Lost Marianne
- The Mystical Bit Flipping
- The Worst Mail Ban
Time Is Not What You Think It Is
Bytes, Characters and Other Betrayals
Decisions Made by Intelligent People
- The .project Dilemma
- The 300 Pages Monitor Manual
- The Giant Gantt
- The Open Source Advantage
- The Partial Scheduler
- The Code That Never Dies
Small Code, Large Blast Radius
Nobody Is Checking Anything
Every one of these systems had validation. On paper.
The 17th Month of the Year Is the Best of All
I always asked developers to use defensive programming and to never trust any data coming from inside or outside our IS. We also try to isolate each partner using dedicated jobs that offer anti-corruption layers. It is extremely important to prevent any invalid data from entering our IS, because it may create technical errors and, once it is in, we have to deal with it.
This time, the data was sent by a partner that exposes a portal to end users and was supposed to perform deep controls on the inputs. Everything worked fine for several years but, at some point, they rewrote their portal (mainly for accessibility and ergonomics requirements) and didn't port the controls they performed previously. Moreover, the portal was designed to allow users to type dates in three integer fields and not with a date picker, because we have to accept partial dates (only the month, or even only the year of some events). The new version was released on a Wednesday at 3 PM and it took only 11 hours to get corrupted data: a user typed 17 in the month field (probably confusing it with the day field). BTW, this is the proof, if needed, that a system should never ever count on humans to enter valid inputs.
This data made our system crash and it was impossible to recover, because each time we restarted it, the corrupted data was read again. How is it possible? Our jobs and APIs were supposed to check any input data! The architecture was: 1) a job that calls the partner API to get the data, controls it and adapts it to our internal model; 2) the job calls one of our internal core APIs to store the data; 3) the API stores the data in a PostgreSQL database.
Fortunately, the corrupted 17 was not stored in the associated NUMBER column. A thorough analysis showed that:
- The data was not controlled by our partner (as stated); they added the controls a few days later.
- The job controlled almost nothing.
- The internal API controlled nothing at its entry point.
BUT:
- By chance, the internal API had to build a date with the incoming data to set it in its response. This date construction crashed and, without it, we would have silently stored dates with month number = 17!
To sum up: no control was coded, but the code that was not supposed to control it did it indirectly!
A Very Clever Check
We had to exchange images using some API and a queue system with a partner. The political relations with the partner were not the best and the partner was very disorganized. We had to make sure to control drastically what they sent us because, once accepted, we had to deal with it, make corrections or reject it manually if it didn't match our IS.
We controlled many things, like the content of the file (with Apache Tika), we ran the antivirus and we accepted only MIME types and extensions in a white list (including png, jpeg, pdf).
The developer coded it with a lot of creativity: instead of rejecting the file if it was not in the list, we checked that the MIME type exposed by the partner request matched the extension of the file. Obviously, any image matched, and a few days after the release we began to receive very unexpected formats like bmp, which we weren't able to display and handle correctly in our web application.
When we fixed it, we had to deal with a stock of corrupted files in our IS that we have to keep indefinitely, and the partner began to receive errors when trying to send files outside the scope of supported images. It created high-level political friction and took years to fix on the partner side.
The ArchUnit Misconfiguration
On a large project, we used a hexagonal/clean architecture and, to enforce it in our Java code, we used ArchUnit (a simple yet powerful Java test tool that controls the dependencies between packages using package name matchers).
On the most important module, we never had any problem reported by this test and we were delighted to see that the developers had truly understood the architecture concepts in place.
However, after 5 years, it appeared that the rules were applied against a myapp package root (the example package). We figured it out because of a stricter ArchUnit enforcement (packages have to exist). Once fixed, we had hundreds of failures and several weeks of rework to fix the architecture...
Ghosts in the Infrastructure
The application was innocent. Something several layers below it was not.
The Credit Card and the UUID
An API returned accounting data for a given item UUID but, on very odd occasions, the API returned a technical error. After weeks of analysis, it turned out that a network security appliance (F5 BIG-IP) had a credit card number leakage prevention feature activated and, when the UUID (in hexadecimal, thus containing letters 'a' to 'f') contained only digits, it was falsely detected as a credit card number and the call was blocked.
The Ballooning Problem
In production, on a large project, we used many virtual machines running several Tomcat JVMs. At random moments, some of them were killed instantly by the system OOM killer, which selects the most memory-intensive apps and kills them to preserve server integrity before a full OOM.
The problem was that we had no idea why this happened. Indeed, we reserved all the maximum memory possible as soon as the VM started, by setting Xms and Xmx properly on the Tomcat JVMs.
After several months of investigation, we learned that ops had overcommitted memory on our ESXi cluster: the sum of the memory allocated to the VMs was larger than the physical memory of the hosts. When a host runs short, ESXi asks the balloon driver (installed in the guest with VMware Tools) to inflate: the driver allocates memory inside the guest and pins it, which forces the guest kernel to reclaim memory elsewhere — normally by swapping the least used pages. The host then reuses the physical pages behind the balloon for another VM.
But instead of quietly swapping, the Linux kernel found itself under sudden memory pressure, looked for the biggest consumer — a Tomcat, of course — and killed it.
The Lost Marianne
Like all French gov websites, ours displayed the official state logo: the national flag with a Marianne — the allegory of the French Republic — over it. The blue and the red come from the flag itself, but the white band is actually the white Marianne drawn on top of it. The asset is provided by a transversal government service that produces the state communication kit, and it was a multi-layer SVG referencing an external image for the Marianne layer.
When going to pre-production, I (luckily) noticed that something was wrong with the flag: it was blue|red instead of blue|white|red — the Marianne was simply missing. What could possibly have happened? A foreign defacement?
Not at all. Our CSP (Content Security Policy, a set of rules telling the browser which resources a page is allowed to load) had been customized a bit too strictly. The img-src directive did not cover the domain the SVG was pulling its Marianne layer from, so the browser silently refused to load it — leaving a flag with a hole in the middle. The fix was to add the missing origin to the policy, and the website was ready for production.
Sometimes, doing things simply, like using a single flat static image, works and avoids many unexpected results...
The Mystical Bit Flipping
One day, an integrator did a copy/paste of a large data file containing many UUID v4. At the application level, a line in the middle of the file of several hundred lines caused an issue. After a long investigation, it appeared that a 'c' had turned into an 'e' (which requires two bit changes).
We never found the origin of the issue, but we suspected a hardware (double) bit flip...
The Worst Mail Ban
This one is partially my fault. On a Wednesday, we opened a very large web application used by 5M people that sends important emails. It was a project we had worked on for 4 or 5 years and it was the big day! I was one of the IT architects and coded some parts, including a template system used to create emails massively and to send them to an SMTP server.
The first emails were not sent in production despite all the tests we had performed! What could possibly have happened?
The cause was a pretty incredible combination of these events:
- Our SMTP server only routed mails to another SMTP server from a partner, because this partner was white-listed and well known.
- The SMTP server of the partner was configured to definitively ban any source IP that sent more than a given amount of emails per minute (which we were largely over). Obviously, our SMTP server IP was white-listed. BUT our SMTP server IP had changed a few days before, due to unrelated infrastructure changes, and this went unnoticed.
- The day before, I performed a last benchmark of my email templating system to reassure myself, and pushed it (successfully) at a rate of several million emails per hour. This benchmark was run in a test environment. It used an internal test SMTP server without any connection to the partner. The configuration was as code, using Puppet (an IaC system that provisions apps on servers), and was configured to force the use of our production SMTP server by default.
- I forced my test server to use a test SMTP server and tried to disable Puppet, but it apparently failed and the target SMTP server was overridden with the production one without me noticing it.
- The day before the release, I thus sent, without noticing, millions of emails to our production SMTP server, which routed them to our partner's one, which banned us quickly because our IP was not in its white list. Nobody noticed it, no alert was sent by our partner and, on the Wednesday, we were already banned before sending our first email!
Time Is Not What You Think It Is
Dates look like the simplest data type there is. They are not.
The Day That Never Comes
Once, we were releasing an important version late on a Friday night. I was still at work at 10 PM and ended up fighting a mystical issue until 3 AM.
At the time, we had the very bad habit of storing a time (00:00:00) together with a date and a timezone, even when what we actually needed was just a date. By the way, this is a very bad idea. It takes slightly more space, but more importantly, it introduces false precision and unnecessary semantics. And using midnight as the default time is particularly dangerous: you are right at the boundary of the previous day, so a timezone conversion can easily move the value to another date. I have seen this happen many times.
But this time, the problem was not a date change. We were working with a large test dataset in which every datetime had its time set to 00:00:00. As a consistency check, our code verified that the time was still exactly midnight, in order to detect data corruption. Every test passed except one: the birth date of a person. I am apparently still traumatized enough to remember his name 15 years later...
The datetime we got was: January 2, 1911 — 00:09:21 (I did not notice it immediately, but this person was also the oldest one in our dataset).
Where on Earth could those 9 minutes and 21 seconds have come from? There was no such constant anywhere in our code. No calculation producing it. Nothing.
After hours of fighting with the problem, I finally understood what was happening. Java uses the IANA timezone database (tzdata), which contains not only current timezone rules but also a remarkable amount of historical timezone information (that goes as far back as the 19th century). When the UTC instant was converted to Europe/Paris, Java applied the historical rule that was valid in France at the time.
Before 11 March 1911, Paris civil time was based on Paris Mean Time, which was 9 minutes and 21 seconds ahead of Greenwich Mean Time. On 11 March 1911, France changed its legal time by subtracting those 9 minutes and 21 seconds, effectively aligning it with Greenwich time. So this completely mysterious 00:09:21 was not a bug, a rounding error, or some forgotten constant buried in our application.
It was astronomy and French history leaking into our Java application through the timezone database!
The Test Failure After 10 PM
I had to release an important version of our project after 10 PM but the build didn't work: a unit test (UT) failed on a time check. I had to wait until the next morning to be able to build the app. What happened?
One of the features of our app was to block usage at night, between 10 PM and 6 AM, for security reasons.
We figured out that some UTs used the current date and time, so the tests only passed during daytime hours! We fixed it by mocking the clock.
What Do You Do on 32 February?
One of the most common cases of mystical issues is the one created by race conditions. On a project, we printed datetimes in logs and some pretty strange dates like 32 February appeared. The root cause was that the code was not thread-safe and shared a single SimpleDateFormat (an old utility class that is not thread-safe)...
Bytes, Characters and Other Betrayals
What comes out of one system is rarely what the next one expects.
Do You Prefer  or �
In French, Â is a perfectly official character and can be used in administrative records. Ã is not, and was not supported by our web application.
We had a data corruption on a user name containing a legitimate Â. After a long investigation, it appeared that the database actually held a à — the result of a classic UTF-8 / Latin-1 charset encoding confusion.
Here is exactly what happened.  is U+00C2, which UTF-8 encodes as the two bytes C3 82. When those bytes are read back as Latin-1 (or CP1252), each one becomes its own character: C3 gives Ã, and 82 gives the discreet ‚ glyph in CP1252. So our  had silently turned into Â.
This is the reason it was so hard to spot: the first character of the pair looks almost exactly like the original one, and the second is nearly invisible. A one-character name corruption that reads, at a glance, as the correct name.
We were very unfortunate that the corrupted character was so similar to the actual one — with almost any other accented letter, someone would have noticed immediately.
The Encoding Nightmare
A legacy system still used nowadays has to upload positional data files as text files downloaded manually from another web application exposed by a partner. The exposed data files use (correctly) a UTF-8 encoding but, for obscure reasons that probably have to do with the fact that the legacy app is a Windows C++ executable, the expected encoding is CP1252 (the default Windows encoding, sometimes called 'ANSI').
One or two lines of code would probably have done the transcoding, but no: users have been asked to manually change the encoding from UTF-8 to CP1252 by opening the file in Notepad++ (a famous open source and powerful Windows text editor), changing the encoding, saving it into another directory and then uploading the new file into the legacy C++ app.
Obviously, the same process in reverse order has to be done to upload data from the C++ app to the partner web application that allows uploading data files!
The Double Base64 Encoded
Another partner exposed a SOAP API and didn't pretend to be RESTful. But it also came with some funny fancies: the data was returned in base64 inside an XML envelope. But when we decoded it, we figured out that the resulting data was still base64! The partner confirmed that the data had to be decoded twice!
The GET /POST Endpoint
We had to work with an API exposed by a partner to get the queries we had to reply to. This was a brand new 'REST' API, just rewritten from an older SOAP one. After digging a bit, it appeared that the 'REST' API was actually just using HTTP verbs to transfer the same XML data as before, but we had a shock when we found out the name of the endpoint to be called to get the data: /GET. And obviously, this endpoint had to be called with a POST, so we had to call: POST /GET!
Decisions Made by Intelligent People
None of these were caused by incompetence. That is exactly what makes them interesting.
The .project Dilemma
This is a good example of a crazy decision based on over-strict requirements. On a very large project containing tens of Eclipse (a famous Java IDE) projects, the architect asked us not to commit the .project metadata files used by Eclipse to build each project. The rationale was to avoid over-dependency on Eclipse (whereas switching to another IDE like IDEA would barely have consisted of deleting the .project and adding an IDEA .iml one).
The solution was to release a script that downloaded all the .project files from a shared network directory whenever a new developer arrived!
The 300 Pages Monitor Manual
On the same project, that not-so-pragmatic client asked us to produce documentation for the over-engineered monitoring system he had asked us to implement. Each application module had tens of monitored values like state of service, number of connections, etc. This extremely brittle documentation was a 300-page PDF with all the possible states and their meaning. It was so big that it was not even possible to print it.
Moreover, we had to produce this documentation in a very short amount of time. A colleague had the idea to use a robot automation scripting system to produce a very mechanical documentation and we released it.
A few days later, the ops who had to use it explained that they never even opened it because they only monitored the system (memory, CPU of the server)...
The Giant Gantt
One of the most ambitious architectures I worked on was a large project that generated planning for the industry. The idea was basically to generate a Gantt diagram of several tens of thousands of task lines over 5 years. Each task could be dragged and dropped to another moment and the entire planning had to be recomputed instantly (less than 2 seconds) following hundreds of business rules (it used a very costly business rules engine). I had to use Java, C++ and SQL (and a lot of other languages) to build this architecture. The requirement was pretty stupid: the customer absolutely wanted to make each task visible over such a long period, only for strategic people and management visibility. They mixed up two different use cases: the operational planning for the workers, for whom a few weeks was largely enough, and the strategic and consolidated planning for the management...
We finally managed to meet the 2-second objective (starting at more than 30 minutes per action when I started to work on optimizations) after 10 months of extreme optimizations, but the project was still mainly a failure because it was over-designed and functionally far too complex to be used. After years of delay and large simplifications and rework, it was finally released and useful.
The Open Source Advantage
On a large project for a company, we were supposed to write our Java app to be run in the WebSphere (IBM) Java application server. I tried to avoid using any specific features of the middleware to keep the possibility of switching to another application server, and we used the very costly (€15K+/CPU) product on all our environments, including development. We asked if a license was paid for us (as subcontractors) in development and we tried several times to convince the client to go to an open source alternative (JBoss AS), for peace of mind and because it worked at least as well as WebSphere, but the customer always replied that everything was under control and that we had to keep using this IBM product.
One day, the customer had an IBM audit (the men in black checking all running instances and comparing them to the paid licenses). It appeared that, contrary to what the customer had told us, the dev environments had to be paid for too. A few days later, they asked us to switch to JBoss without explaining why (it was long after this request that we became aware of the audit).
The Partial Scheduler
On another project for a retirement administration, we had to schedule a large number of jobs. The customer used a very complex and costly product (Control-M) while their budget was restricted. One day, in production, the customer informed us that some data was outdated. After investigation, it appeared that some jobs had not been run. We looked at the call tree between jobs and we saw that, in production, it contained only a portion of the jobs supposed to run.
After much investigation, it appeared that, since the customer was limited in maximum number of jobs, he used a scheduler in the DEV environment to run the remaining jobs — and that environment was down...
The Code That Never Dies
I once worked for an organization that (at this time at least) never rewrote its code. They preferred to write wrappers, conversion tools, facade patterns and so on. The most extreme case I witnessed — and had to design — was to expose a large number of TPR screens behind SOAP services. TPR screens are a legacy French mainframe technology (from Bull) that was used to display transactional screens in character mode. It was used to display or type in some values. The idea was to keep all this TPR as it was, to screen-scrape the result screens for read SOAP services and to simulate the typing of values at some locations of the screen for write-oriented services.
The most incredible thing about it is that it worked pretty well and was not so slow!
Small Code, Large Blast Radius
A few lines, a misunderstood default, and the whole production is on fire.
The Scope Issue
On a large webapp, we coded a classic error banner displayed on screen on error. One day, the IT director called us in a hurry because of a very severe issue in production following an update: each time an error occurred for an agent, the same banner appeared on all agents' screens!
After a quick investigation, the reason was pretty obvious: the error management used a Spring bean and Spring uses the singleton scope by default. It means that there is a single instance of this class for the whole running application. So when an error occurred, it updated the shared state of the banner message and thus all agents saw the same message. It is a very simple way to explain the importance of understanding the scope concept to developers. In this case, they should not have kept request state in a shared bean at all: the error message belongs to the response, so it should have been passed through the model returned to the view.
Don't Forget the Padding!
A senior dev had to write a purge job for important data, in order to comply with the GDPR. The data was stored in S3 buckets named <month number>-<bucket type>, with a zero-padded month: 01-bills, 02-bills, and so on.
To find the bucket of a given month, the code read the month number as a string and kept only its first character, then padded it back to two digits. It worked perfectly for eleven months a year. But for October, "10" became "1", which was padded back into "01" — so the job happily started deleting the bills of January while believing it was purging October.
It could have been a major data loss, followed by a long and painful restoration. We were saved by a second bug: the purge was written so inefficiently that it was still crawling through the first objects when we noticed the problem. Not a single file had actually been deleted.
Two bugs in the same job, and the bad one cancelled out the worse one!
The Random Authentication Failure
In a service written in Node.js (thus JavaScript), we wrote a system that automatically created passwords for upcoming exchanges with a partner. But around every 10K exchanges, we had a strange syntax error.
It appeared that when a generated password contained more than a single '$', the JS framework wasn't able to escape it properly.
The Human Factor
Sooner or later, there is a person at the end of the chain.
The Romantic Bug
We set up an electronic signature system for administrative documents and one day a bug was filed: the electronic signature of some documents was different from the handwritten signature image on the document. One was a man's and the other a woman's.
It appeared that they were actually a couple and shared the same house. They had swapped their corporate laptops without noticing...
After this, we added a control to ensure the match between the app login and the smart card.
Chown of the Dead
During my final-year engineering internship (5 months) in a software company, I spent a substantial amount of that time configuring and documenting a new Unix (AIX) development server. There was no packaging system or IaC system in the late 90s. On the last day of my internship, I advised my boss never to give the root password to developers. A few days later (I had returned to my engineering school for the last courses), he called me, explaining that a developer had run chown -R foo $MYDIR/ as root and that the server was no longer usable because of rights issues. I think you have guessed the value of $MYDIR...
The Mechanical Reboot
In a small and crappy datacenter, an IT department had to reboot a server daily (due to a memory leak) but had lost access to it. They decided to schedule an eject command on the CD tray of another server, taped a stick on it and positioned it so that the stick pressed the power button of the server to be restarted, and did it again a few seconds later to start it back up.
Conclusion
When I remember them, I smile at many of these issues, but at the time they occurred, they caused us a lot of pain. But this proves that pain is the best way to remember things. However, most of the people who made these errors probably forgot about the issue and never learned from it. I did, just because I was the architect facing the client and the hierarchy and, above all, because I was an integrator who had to deal with production. If you are a developer, I invite you to read this other article that presents the things a developer must know about the production world. On a positive side, many of these issues are a very good source of inspiration for my IT architecture courses.