{"authors":[{"name":"Kyle George"}],"description":"Kyle George's blog and notes.","favicon":"https://kylegeorge.com/favicon.ico","feed_url":"https://kylegeorge.com/feed.json","home_page_url":"https://kylegeorge.com/","items":[{"authors":[{"name":"Kyle George"}],"content_html":"\u003cp\u003e\u003cmark\u003eThere are no transient issues, only issues you don\u0026rsquo;t understand yet.\u003c/mark\u003e\u003c/p\u003e\n\u003cp\u003eIf you work in technology you\u0026rsquo;ve no doubt seen numerous problems that were\ndismissed with a hand wave and a \u003ccode\u003etransient issue\u003c/code\u003e or \u003ccode\u003etransient error\u003c/code\u003e\ncomment.  \u003cem\u003e\u0026ldquo;Closed, transient issue.\u0026rdquo;\u003c/em\u003e   \u003cem\u003e\u0026ldquo;Resolved, transient error.\u0026rdquo;\u003c/em\u003e\nFlapping alert?  \u003cem\u003eTransient\u003c/em\u003e.  Fixed by restarting service?  \u003cem\u003eTransient\u003c/em\u003e.\nExcept that there\u0026rsquo;s no such thing.  There is no such thing as a transient\nissue, there are only issues that you don\u0026rsquo;t understand yet.\u003c/p\u003e\n\u003cp\u003eThese supposed \u003cem\u003etransient issues\u003c/em\u003e are telling you something.  Are you\nlistening?  That flapping alert might be telling you that your system can\u0026rsquo;t\nhandle peak load and could fall over with a spike in traffic.  That problem\nthat was fixed by restarting that service could be masking a memory leak.\nThat periodic \u003ca href=\"https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429\" rel=\"noopener\"\u003eHTTP\n429\u003c/a\u003e\nerror you get from the \u003ca href=\"https://docs.aws.amazon.com/ec2/latest/devguide/ec2-api-throttling.html\" rel=\"noopener\"\u003eAWS\nAPI\u003c/a\u003e\ncould be telling you that you\u0026rsquo;re one scale-up away from a widespread outage.\nThat \u003cem\u003etransient\u003c/em\u003e 429 is telling you that your call rate is periodically\nexceeding the \u003ca href=\"https://en.wikipedia.org/wiki/Token_bucket\" rel=\"noopener\"\u003etoken bucket and refill\nrate\u003c/a\u003e, and your system is flying\ntoo close to the sun.  But hey, at least your KPIs are green because you\nresolved that alert within the SLA.\u003c/p\u003e\n\u003cp\u003eTransient errors are signals, but it\u0026rsquo;s easy to ignore or paper over them.\nGoogle\u0026rsquo;s AI Overview for \u003ccode\u003etransient error\u003c/code\u003e is \u0026hellip;\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eA transient error (or transient fault) is a temporary, self-correcting\nfailure caused by brief disruptions like momentary network drops, a busy\nservice, or a temporary resource lock. It resolves itself quickly without\nneeding a permanent fix.\u003c/p\u003e\n\n\u003c/blockquote\u003e\n\u003cp\u003e\u0026hellip; and it dutifully links to a \u003ca href=\"https://learn.microsoft.com/en-us/azure/architecture/best-practices/transient-faults\" rel=\"noopener\"\u003ewall of text from\nMicrosoft\u003c/a\u003e\npreaching about retries, retry strategies, exponential backoff, etc.  Sure,\ngood engineering requires that you consider these things as part of the\n\u003ca href=\"https://en.wikipedia.org/wiki/End-to-end_principle\" rel=\"noopener\"\u003eend-to-end principle\u003c/a\u003e\nof system design.  Yes, you do need things like intelligent retries.  But\nthese are things that get you through the night, they aren\u0026rsquo;t meant to be in\nuse perpetually.  If your software is retrying operations all the time you\u0026rsquo;re\nmissing a cry for help.  Are your retries instrumented?  Do you alert on them?\u003c/p\u003e\n\u003cp\u003eI completely understand that during a busy on-call shift the flapping alert\ngets closed in favor of giving the cycles to higher immediate priorities.\nTeams must have a process for individuals or managers to recognize patterns or\nmetrics that deserve greater attention.\u003c/p\u003e\n\u003cp\u003eMany years ago after observing more than one \u0026ldquo;resolving, transient error\u0026rdquo;\nacross multiple on-call shifts, only for that warning sign to later blow up\ninto a bigger problem, I added a Slackbot autoresponder at my company.  Anyone\nwho sent messages containing \u003cem\u003etransient error\u003c/em\u003e or \u003cem\u003etransient issue\u003c/em\u003e would get\na friendly reminder:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cmark\u003eThere are no transient issues, only issues you don\u0026rsquo;t understand yet.\u003c/mark\u003e\u003c/p\u003e\n\n\u003c/blockquote\u003e\n\u003cp\u003eThat helped, but it wasn\u0026rsquo;t enough on its own.  I created a wiki page where\npeople could put known issues.  I called it\n\u003ca href=\"https://en.wikipedia.org/wiki/NOTAM\" rel=\"noopener\"\u003eNOTAMs\u003c/a\u003e, or NOtice To All Members,\ninspired by aviation\u0026rsquo;s use of the same technique.\u003csup id=\"fnref:1\"\u003e\u003ca href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\"\u003e1\u003c/a\u003e\u003c/sup\u003e  Team members were\nasked to check this at the start of their shift, and record any notable\nanomalies.  Finally, we backstopped the process with alert review, on-call\nshift handoff procedures, and encouraging people to log tickets for follow-up\nwhen they couldn\u0026rsquo;t immediately give it time.  Overall, I\u0026rsquo;m happy to say the\ncombination of these techniques greatly reduced the missed occurrences of\n\u003cem\u003etransient issues\u003c/em\u003e blowing up in our faces.\u003c/p\u003e\n\u003cp\u003eHere\u0026rsquo;s an instance \u0026mdash; pun intended \u0026mdash; where it paid off.  Multiple jobs\nacross different parts of our environment were having elevated retries for the\nEC2 API \u003ccode\u003eDescribeInstances\u003c/code\u003e.  Runtimes were delayed (as is frequently a\nsymptom of retries) but the system was still operating.  Pattern recognition\nand having a culture that values understanding transient issues led us to\ndiscover that we were exceeding our \u003ccode\u003eDescribeInstances\u003c/code\u003e rate limit within a\nsingle account, and we were able to take a combination of actions to request a\nrate limit increase and reduce our utilization (caching).  If we had waited\nmuch longer the retries would have caused time-sensitive systems to fail.  By\nrecognizing the \u003cem\u003etransient\u003c/em\u003e pattern, wanting to understand it, and taking\ntimely and decisive action, we were able to head off disaster.\u003c/p\u003e\n\u003cp\u003eThese days with AI agents and anomaly detection in open source and vendor\nmonitoring products there are a lot more options at your disposal.  You can\npoint your agent at, say, a month\u0026rsquo;s worth of alerts and ask it to find the\npatterns that you missed.  Whether you\u0026rsquo;re doing it with human eyes or AI\naugmentation, the point is to do something.\u003c/p\u003e\n\u003cp\u003eIn summary:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eTransient issues are a chance to catch problems before they become severe.\nDon\u0026rsquo;t waste the opportunity.  Understand them and take action on transient\nissues before they blow up.\u003c/li\u003e\n\u003cli\u003eInstrument your retries and error handling and alert on their overuse.\nTeach operators that repeated manual actions aren\u0026rsquo;t normal and to surface\nanomalies.\u003c/li\u003e\n\u003cli\u003eBuild a culture and put a process in place to recognize patterns and\nrecurrences and not lose track of them.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eWhat do you have to learn from your \u003cem\u003etransient issues\u003c/em\u003e?\u003c/p\u003e\n\u003cdiv class=\"fine-print\"\u003e\n\u003cp\u003eAI Use Statement: I certify these are 100% my words and ideas.  AI was used\nonly as a proofreader.\u003c/p\u003e\n\n\u003c/div\u003e\n\n\u003cdiv class=\"footnotes\" role=\"doc-endnotes\"\u003e\n\u003chr\u003e\n\u003col\u003e\n\u003cli id=\"fn:1\"\u003e\n\u003cp\u003eThey call it \u003cem\u003eNotice to Airmen\u003c/em\u003e or more recently \u003cem\u003eNotice to Air\nMissions\u003c/em\u003e.\u0026#160;\u003ca href=\"#fnref:1\" class=\"footnote-backref\" role=\"doc-backlink\"\u003e\u0026#x21a9;\u0026#xfe0e;\u003c/a\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003c/div\u003e","date_modified":"2026-10-09T20:24:56-04:00","date_published":"2026-10-09T18:45:00-04:00","id":"https://kylegeorge.com/blog/no-transient-issues/","summary":"There are no transient issues, only issues you don’t understand yet.\nIf you work in technology you’ve no doubt seen numerous problems that were dismissed with a hand wave and a transient issue or transient error comment. “Closed, transient issue.” “Resolved, transient error.” Flapping alert? Transient. Fixed by restarting service? Transient. Except that there’s no such thing. There is no such thing as a transient issue, there are only issues that you don’t understand yet.","tags":["Engineering Culture","Reliability Engineering","Engineering","Observability","On-Call","Troubleshooting"],"title":"There's No Such Thing as a Transient Issue","url":"https://kylegeorge.com/blog/no-transient-issues/"},{"authors":[{"name":"Kyle George"}],"content_html":"\u003cdiv class=\"profile\"\u003e\n  \u003cspan class=\"author-photo profile__photo\" style=\"--zoom: 1; --dx: 0%; --dy: 0%\"\u003e\u003cimg src=\"https://kylegeorge.com/977445_2278581154595038478_hu_b2c45311b9a3a01a.webp\" width=\"296\" height=\"296\" alt=\"Photo of Kyle George\"\u003e\u003c/span\u003e\n  \u003cdiv class=\"profile__text\"\u003e\n    \u003cp class=\"profile__name\"\u003eKyle George\u003c/p\u003e\u003cp class=\"profile__description\"\u003eme\u003c/p\u003e\n  \u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003eWell it\u0026rsquo;s \u003ca href=\"https://kylegeorge.com/about/\"\u003eabout\u003c/a\u003e time I set this up.  I\u0026rsquo;ve only been toying\nwith the idea for 15+ years \u0026#x1f604;.  Better late than never.\u003c/p\u003e","date_modified":"2026-09-28T02:25:05-04:00","date_published":"2026-09-28T00:26:31-04:00","id":"https://kylegeorge.com/blog/hello-world/","summary":"Kyle George\nme\nWell it’s about time I set this up. I’ve only been toying with the idea for 15+ years 😄. Better late than never.","title":"Hello, world!","url":"https://kylegeorge.com/blog/hello-world/"}],"language":"en-US","title":"Kyle A. George","version":"https://jsonfeed.org/version/1.1"}