Two security items today, both worth checking against your own fleet rather than waiting for the next patch cycle, plus an AWS monitoring feature that quietly kills a homegrown workaround a lot of us are running.
\nCVE-2026-65400, a bug in macOS's built-in Screen Sharing (VNC over TCP 5900) that let a network attacker connect without valid credentials, is now under active exploitation. The Netherlands' NCSC says it's seen abuse on multiple systems with port 5900 open to the internet — in every case the attacker landed root and dropped a Monero miner (BleepingComputer). Apple actually fixed this back on August 6 in macOS Tahoe 26.6.1, Sequoia 15.7.9, and Sonoma 14.8.9, but patch uptake on Macs sitting outside MDM is always spotty, and this is exactly the kind of service that gets enabled once for a support session and never turned off. If you manage any Macs — dev workstations, kiosk boxes, build machines — check whether Screen Sharing is on (System Settings → General → Sharing) and either disable it or confirm the update is installed. A crypto miner is the visible symptom; unauthenticated root access is the actual problem.
\nResearcher Nightmare Eclipse published a technique called ShieldBreak that defeats a fix Microsoft shipped for an earlier Defender flaw (RoguePlanet), using a user-mode callback hook to tamper with file contents mid-scan via the Cloud Filter API (BleepingComputer). It affects Windows 10, Windows 11 25H2, and Windows Server, the PoC reportedly hits 100% on tested systems, and there's no CVE or patch yet — Microsoft says it's still investigating. It requires Defender to be enabled and running, so this isn't remote-code-execution-from-nothing, but it is a working local privilege-escalation path with no fix on the horizon. Nothing to patch today, but worth flagging to whoever owns endpoint security on your Windows fleet so it's on their radar before a patch lands, not after.
\nAmazon EC2 now has Application Status Checks — instance-level monitoring that periodically hits an HTTP/HTTPS port and path you define and confirms it returns the response code you expect, on a 60-second interval (AWS). It sits alongside the existing infrastructure status checks and plugs straight into Auto Scaling, which can replace an instance whose app-level check fails even if the instance itself looks perfectly healthy — a stopped web server, a crashed Docker daemon, a broken app dependency. If you've ever bolted a sidecar or a custom CloudWatch alarm onto an ASG just to catch \"the instance is up but the app is dead,\" this replaces that homegrown setup with a native, tag-or-instance-ID-scoped check. Available in every commercial region plus GovCloud, no extra infrastructure required.
\nAWS Direct Connect has had degraded connectivity since 13:17 UTC yesterday for customers connected at the Equinix FR5 facility in Frankfurt — a co-location partner facility issue, not an AWS-side fault, with AWS recommending VPN failover in the interim. If you've got a DX connection through FR5 and no redundant path through another location, that's worth checking on today.
\nNeither security item needs a fire drill, but both are worth five minutes: confirm Screen Sharing is off or patched on any internet-reachable Mac, and make sure Windows Defender's ShieldBreak gap is on your endpoint team's watchlist. The EC2 health check feature is the one to actually go try — it's a straightforward win if you're currently monitoring app health with anything homegrown.
\n", "date_published": "2026-08-16T00:00:00.000Z", "tags": [ "roundup", "aws", "security" ] }, { "id": "https://jstgtech.com/blog/2026-08-16-vpc-endpoints-nat-gateway-cost/", "url": "https://jstgtech.com/blog/2026-08-16-vpc-endpoints-nat-gateway-cost/", "title": "Cutting NAT gateway costs with VPC endpoints that actually help", "summary": "How gateway and interface VPC endpoints replace NAT gateway traffic for AWS API calls, what they cost instead, and which traffic still has to go through NAT.", "content_html": "A NAT gateway bills $0.045/hour just to exist (about $32/month per AZ) plus\n$0.045/GB processed, in every region, for every private subnet that needs\noutbound internet access. Most of that traffic, on a typical workload, isn't\nactually going to the internet — it's a Lambda function calling S3, an ECS\ntask writing to DynamoDB, or an EC2 instance pulling a Secrets Manager value,\nall routed out through NAT and back into AWS because nothing told the VPC a\nshorter path existed. VPC endpoints are that shorter path. They don't replace\nNAT gateways outright, but on most accounts they eliminate the majority of\nthe traffic NAT was ever pushing.
\n\"VPC endpoint\" covers two unrelated implementations that happen to share a\nname:
\nThe pricing gap is why the order of operations matters: turn on gateway\nendpoints for S3 and DynamoDB first, always — there's no cost trade-off to\nevaluate, they're strictly free traffic that used to route through paid NAT.\nInterface endpoints are the ones that need a per-service cost comparison\nbefore you add them.
\nGateway endpoints attach to specific route tables, not the whole VPC, so\nprivate subnets need the association explicitly:
\naws ec2 create-vpc-endpoint \\\n --vpc-id vpc-0123456789abcdef0 \\\n --service-name com.amazonaws.us-east-1.s3 \\\n --route-table-ids rtb-0a1b2c3d4e5f6a7b8 rtb-1a2b3c4d5e6f7a8b9 \\\n --vpc-endpoint-type Gateway\nIn Terraform:
\nresource \"aws_vpc_endpoint\" \"s3\" {\n vpc_id = aws_vpc.main.id\n service_name = \"com.amazonaws.us-east-1.s3\"\n vpc_endpoint_type = \"Gateway\"\n route_table_ids = [for rt in aws_route_table.private : rt.id]\n}\nCheck that it actually took effect with a route table describe — you're\nlooking for a pl- (prefix list) destination pointing at the endpoint's ID,\nnot a 0.0.0.0/0 route to NAT for S3 traffic:
aws ec2 describe-route-tables --route-table-ids rtb-0a1b2c3d4e5f6a7b8 \\\n --query 'RouteTables[0].Routes[?DestinationPrefixListId!=`null`]'\nInterface endpoints need a subnet placement (one ENI per AZ you list) and a\nsecurity group, since they're a real network interface that other resources\nconnect to:
\nresource \"aws_vpc_endpoint\" \"secretsmanager\" {\n vpc_id = aws_vpc.main.id\n service_name = \"com.amazonaws.us-east-1.secretsmanager\"\n vpc_endpoint_type = \"Interface\"\n subnet_ids = aws_subnet.private[*].id\n security_group_ids = [aws_security_group.vpc_endpoints.id]\n private_dns_enabled = true\n}\n\nresource \"aws_security_group\" \"vpc_endpoints\" {\n vpc_id = aws_vpc.main.id\n ingress {\n from_port = 443\n to_port = 443\n protocol = \"tcp\"\n security_groups = [aws_security_group.app.id]\n }\n}\nprivate_dns_enabled = true is what makes this transparent to application\ncode: the SDK still calls secretsmanager.us-east-1.amazonaws.com, but\nRoute 53 Resolver answers with the endpoint's private IP instead of the\npublic one, inside the VPC. Nothing in your application config changes —\nwhich is also why it's easy to add one and not notice it's doing nothing,\ncovered below.
VPC endpoints only cover AWS service APIs that have a PrivateLink or gateway\nimplementation. NAT (or an internet gateway with a public IP, for public\nsubnets) is still required for:
\nDon't decommission your NAT gateway because you added endpoints for your\ntop three services — audit what's actually calling out first (see below),\nbecause the leftover traffic is usually smaller but never zero.
\nDon't add endpoints speculatively for every service AWS offers — at\n$0.01/hour/AZ each, a dozen unused interface endpoints across 3 AZs is\n$26/month for nothing. VPC Flow Logs tell you what's actually flowing\nthrough NAT right now. Enable them on the NAT gateway's ENI (or the whole\nVPC) and query the destination:
\nfields dstAddr, bytes\n| filter srcAddr like /^10\\./\n| stats sum(bytes) as totalBytes by dstAddr\n| sort totalBytes desc\n| limit 20\nCross-reference the top destination IPs against AWS's published IP address\nranges for your region —\njq '.prefixes[] | select(.region==\"us-east-1\") | .service' ip-ranges.json\ngives you the service name per CIDR block. If a destination IP resolves to\nDYNAMODB or S3, that traffic is a candidate for a free gateway endpoint\nright now. If it resolves to SECRETSMANAGER or ECR and represents a\nmeaningful share of your NAT bytes total, run the $0.01/GB endpoint cost\nagainst what that GB volume currently costs at NAT's $0.045/GB — the\nendpoint usually wins by a wide margin, but a low-traffic service isn't\nworth the flat hourly charge across every AZ, since that's a fixed cost\nyou pay whether or not it's used, unlike NAT's per-GB-only marginal cost for\nthat same traffic.
An interface endpoint with private DNS enabled changes DNS resolution, but\nonly for resolvers that actually query the VPC's Route 53 Resolver. Three\nways this silently fails to redirect traffic, leaving you paying for both\nthe endpoint and unchanged NAT usage:
\n.2 resolver — the private DNS record\nnever gets seen, so the SDK still resolves the public IP and routes out\nthrough NAT anyway.https://s3.dualstack.us-east-1.amazonaws.com or a client explicitly\nconfigured with a non-default endpoint) bypasses the standard hostname\nthe private DNS record matches.Confirm it's actually working, don't assume it: dig the service hostname\nfrom inside a resource in the private subnet and check the answer is a\nprivate (10.x/172.16.x/192.168.x) address, then watch the endpoint's own\nCloudWatch metrics (BytesProcessed on the interface endpoint) climb while\nthe NAT gateway's BytesOutToDestination correspondingly flattens. If the\nendpoint's traffic metric stays near zero after deployment, something in\nthe DNS resolution path above is still routing around it.
Turn on S3 and DynamoDB gateway endpoints everywhere, immediately — they're\nfree and there's no scenario where they make things worse. For everything\nelse, pull a week of VPC Flow Logs against the NAT gateway, resolve the top\ndestination IPs against AWS's IP ranges, and add interface endpoints only\nfor the services showing real GB volume, checking each one actually took\nover the traffic afterward rather than assuming the Terraform apply was\nenough. On a typical account this drops NAT gateway processing charges\nsharply without touching NAT's role for genuine third-party internet\ntraffic, which is the traffic it was actually built for.
\n", "date_published": "2026-08-16T00:00:00.000Z", "tags": [ "aws", "vpc", "networking", "cost", "tutorial" ] }, { "id": "https://jstgtech.com/blog/2026-08-14-cloud-roundup/", "url": "https://jstgtech.com/blog/2026-08-14-cloud-roundup/", "title": "Cloud roundup: S3 finally names the policy that denied you", "summary": "AWS S3 access-denied errors now name the exact policy ARN, Client VPN gets a scriptable CLI, and OpenAI ships authorized offensive-security models on Bedrock.", "content_html": "Quieter news day, no fresh KEV entries or breaking CVEs in the last 24 hours — so today's roundup is AWS-only, and it's a genuinely useful batch: a debugging quality-of-life fix, an infra-as-code-friendly VPN client, and Bedrock picking up dedicated offensive-security models.
\nAmazon S3 now includes the specific IAM or AWS Organizations policy ARN in HTTP 403 responses for explicit-deny cases — across SCPs, RCPs, identity-based policies, session policies, and permission boundaries (AWS). If you've ever gotten a bare \"Access Denied\" and had to manually walk every SCP and permission boundary in the chain to find the one deny statement, this removes that entire step — the error just tells you which policy did it. No opt-in, no cost, live in every region including GovCloud and China. This is the kind of change that saves real debugging time without anyone having to change a line of code.
\nThe Client VPN desktop app got a rebuild (v6.0.x, OpenVPN3-based) that adds a CLI with full feature parity to the GUI, so VPN connections can be scripted into CI pipelines and IaC workflows instead of requiring a human to click through the app (AWS). It also adds admin controls to scope VPN profiles to specific users or push a global profile to every device, plus faster connection establishment on Windows, macOS, and Linux. It's a free upgrade, backward-compatible with existing endpoints. Worth grabbing if you've ever had to walk a remote contractor through manually configuring VPN client settings — that's now something you can push as a managed profile instead.
\nAWS added two purpose-built cybersecurity models to Bedrock: Daybreak Blue (GPT-5.6 Sol) for defensive workflows like vulnerability discovery, detection engineering, and incident response, and Daybreak Red (GPT-5.6 Cyber) for advanced authorized work — vulnerability research, exploit reproduction, mitigation development (AWS). Access isn't self-serve — it requires enrolling in Daybreak access through OpenAI or your AWS account team, and it's US East (N. Virginia) only for now. Inference data isn't used for training and isn't shared back to OpenAI by default. If your security team has been prototyping LLM-assisted detection engineering or authorized pentest tooling with general-purpose models, this is worth evaluating once you can get enrolled — it's a narrower, more accountable access model than \"give the whole team an API key.\"
\nNothing urgent to patch today. The S3 error message change is the one to actually notice — it'll quietly save you time the next time you're chasing down a deny across a dozen SCPs.
\n", "date_published": "2026-08-14T00:00:00.000Z", "tags": [ "roundup", "aws", "security" ] }, { "id": "https://jstgtech.com/blog/2026-08-14-amazon-cognito/", "url": "https://jstgtech.com/blog/2026-08-14-amazon-cognito/", "title": "Service spotlight: Amazon Cognito user pools vs identity pools", "summary": "Amazon Cognito is two different services wearing one name — user pools for authentication, identity pools for AWS credentials — and the pricing tiers that trip teams up.", "content_html": "Half the confusion I've seen around Cognito isn't about how it works —\nit's that \"Cognito\" is the marketing name for two separate services that\nhappen to share a console tab. Get the two confused and you either build\nan auth flow that can't touch AWS resources, or wire up AWS credentials\nfor users who were never actually authenticated.
\nUser pools are the authentication half: a managed user directory that\nhandles sign-up, sign-in, password policies, MFA, and email/SMS\nverification, and issues JWTs (ID, access, and refresh tokens) on\nsuccessful login. They can front their own hosted UI or federate to\nsocial IdPs and SAML/OIDC providers, so \"login with Google\" and\n\"login with your corporate Okta\" both terminate in the same user pool.
\nIdentity pools (Cognito Federated Identities) are the authorization\nhalf: given a token — from a Cognito user pool, a social IdP, SAML, or\neven an unauthenticated \"guest\" request — an identity pool exchanges it\nfor temporary AWS credentials via STS, scoped by an IAM role. That's\nthe piece that lets a mobile app upload directly to S3 or call DynamoDB\nwithout a backend in the middle, using credentials that expire instead of\na long-lived API key baked into the app.
\nThe two compose but aren't interchangeable: a user pool alone gives you\n\"who is this person,\" not \"what can they touch in AWS.\" An identity pool\nalone will happily hand out credentials to unauthenticated guests if you\nlet it — it doesn't do authentication itself, it just brokers whatever\ntoken you hand it.
\n${cognito-identity.amazonaws.com:sub} in the\nIAM policy) instead of routing every upload through an API Gateway +\nLambda pair whose only job is forwarding bytes.The tier structure is the sharpest edge. Cognito's current pricing splits\nuser pools into Lite, Essentials, and Plus feature plans, and features\nthat look like they should be table stakes — SAML/OIDC federation,\nadvanced security features like compromised-credential and\nrisk-based adaptive authentication — only exist on Essentials or Plus,\nbilled per monthly active user (MAU). A team that prototypes on Lite\nbecause the free allowance looks generous, then adds \"sign in with our\nIdP\" for an enterprise customer, discovers that single feature moves the\nentire pool's MAU count onto the paid tier, not just the new users using\nit. Check which tier a feature needs before you promise it in a sprint.
\nThe other recurring mistake is treating identity pool credentials as a\nsubstitute for real authorization. STS credentials scoped to an\nidentity-pool IAM role are still full IAM credentials — if the attached\nrole is too broad, an authenticated (or worse, guest-enabled)\nunauthenticated identity can reach far more than \"upload your own\nprofile picture.\" Audit identity pool roles the same way you'd audit any\nIAM role handed to untrusted clients, with Condition blocks scoping\naccess to the caller's own identity ID, not just a wide S3 prefix.
Before adding a Cognito feature to a design doc, check the feature plan\ncomparison for the tier it\nactually requires — \"advanced security\" and third-party federation are\nthe two that most often get assumed as free defaults and aren't. And if\nyou're issuing AWS credentials via an identity pool, run aws sts\nget-caller-identity with a test unauthenticated identity's credentials\nbefore shipping, to confirm guest access is scoped as tightly as you\nthink it is rather than as tightly as you meant it to be.
In March 2019, an attacker used a server-side request forgery (SSRF) bug\nin a misconfigured web application firewall to reach a single internal\nURL — http://169.254.169.254/ — running behind a Capital One-hosted\napplication in AWS. That one request was enough to pull temporary IAM\ncredentials for a role with broad S3 read access, and from there exfiltrate\nmore than 100 million credit applications and 140,000 Social Security\nnumbers (DOJ indictment). No malware, no phishing, no zero-day in AWS\nitself — just a web app that could be tricked into fetching an internal\nURL on the attacker's behalf, and an internal URL that handed over live\ncredentials to anyone who asked. Capital One paid an $80 million OCC fine\nand settled a $190 million class action over it (Reuters). Seven years\nlater, IMDSv1 — the version that made this possible — is still the\ndefault reachable endpoint on plenty of EC2 instances, because \"enabled\"\nis not the same as \"enforced.\"
Every EC2 instance can reach a link-local address, 169.254.169.254,\nwhich serves the Instance Metadata Service (IMDS) — instance ID, AMI\ninfo, user-data, and critically, temporary security credentials for\nwhatever IAM role is attached to the instance. It's how an EC2 instance\ngets AWS credentials without anyone hardcoding a key. In the original\nversion of that service, IMDSv1, the endpoint answers any plain HTTP GET\nrequest with no authentication step at all — if a request reaches that\nIP from the instance, it gets an answer. That design was fine as long as\nnothing running on the box could be tricked into making arbitrary\noutbound requests on an attacker's behalf. But web applications routinely\ndo exactly that: fetch a URL from a query parameter, proxy a request,\nrender a remote image, validate a webhook. The former engineer charged in\nthe case had reportedly discovered a misconfigured ModSecurity WAF rule\nin front of a Capital One web application that could be coerced into\nrequesting an arbitrary URL and returning the response — a textbook\nSSRF (indictment). Point that SSRF at\n169.254.169.254/latest/meta-data/iam/security-credentials/<role-name>\nand IMDSv1 hands back an access key, secret key, and session token for\nwhatever role the instance was running as — no extra proof that the\nrequest came from a legitimate, non-hijacked process.
The stolen role's permissions, not the SSRF bug itself, set the ceiling\non the damage. In this case the role had read access to S3 buckets used\nfor a range of Capital One's credit-card and loan-application data,\ncompiled from six years of applications going back to 2005 — 106 million\nindividuals across the US and Canada, including Social Security numbers,\nbank account numbers, and credit scores (Capital One's disclosure).\nStolen IMDS credentials are functionally indistinguishable from\nlegitimate ones to everything downstream — CloudTrail logs the calls as\ncoming from the role, IAM enforces exactly the policy attached to that\nrole, and nothing about the request signature reveals it originated from\nan SSRF payload rather than the application itself. That's the general\nshape of every IMDS-credential-theft incident: the SSRF or RCE is the\nentry point, but the actual damage is bounded entirely by how\nover-permissioned the instance role was. A role scoped to one bucket\nprefix limits an attacker to that prefix; a role with s3:* across the\naccount hands over the account's data.
AWS's direct answer, shipped in November 2019 a few months after this\nbreach became public, is IMDSv2 — it requires a session to be\nestablished with a PUT request that returns a token, and every\nsubsequent metadata request must carry that token in a header\n(AWS).\nCritically, IMDSv2's PUT requires HTTP with no redirects followed and\nsets a configurable hop limit (default 1) on the response's TTL — which\nmeans a proxy or SSRF payload relaying a simple GET typically can't\ncomplete the handshake or forward the token past one network hop, while a\nprocess running natively on the instance can. That single change would\nhave broken the Capital One attack path as described. Enforcement is the\npart teams skip: IMDSv2 has been opt-in at the instance level since\nlaunch, so an unpatched fleet stays exploitable indefinitely unless\nsomeone acts. Three concrete steps: set HttpTokens: required (not just\noptional) on every instance and in every launch template, ideally via\nan AWS Config rule (ec2-imds-v2-check) that flags drift; use an\n**SCP or Config rule to deny new instance launches without IMDSv2\nenforced**, since a one-time fleet fix doesn't stop the next Terraform\napply from reintroducing IMDSv1; and independently, scope every instance\nrole to least privilege — the hop-limit defense is layer two, not a\nreplacement for making sure a stolen credential is only ever worth as\nmuch as the narrowest policy you could get away with attaching.
This wasn't an AWS vulnerability — IMDS behaved exactly as designed, and\nthe WAF misconfiguration was Capital One's, not Amazon's. But it's the\ncanonical case study for why cloud metadata endpoints deserve the same\nscrutiny as any credential store: anything that can make an outbound\nHTTP request from inside your VPC is a potential path to\n169.254.169.254, and IMDSv1's design assumed that path would never\nexist. IMDSv2's token-and-hop-limit model closes the SSRF-relay case\nspecifically, but it only helps the instances where someone actually\nflipped HttpTokens to required — which is why the Config rule and the\nSCP matter as much as the setting itself. If your fleet still allows\nHttpTokens: optional, the difference between \"we have IMDSv2 available\"\nand \"we're actually protected\" is one unenforced launch template away\nfrom mattering.
The big story today isn't a new breach — it's the bill coming due on an old one. A researcher got hold of the actual haul from March's LiteLLM supply-chain attack, and the scale is worse than the original disclosure suggested. On the AWS side, two changes worth planning around: more control over your EKS control plane, and a pricing shift that quietly doubles some OpenSearch costs.
\nBack in March, attackers compromised the Trivy scanner and used it to slip two malicious versions of LiteLLM onto PyPI for about 40 minutes — long enough for a .pth file to start harvesting secrets from every CI run that installed it. Hudson Rock has now obtained and analyzed the actual stolen archive: 153GB, 433,909 files, with 118,829 CI runner dumps attributable to 2,488 corporate domains — including AWS, Cisco, Samsung, Salesforce, Microsoft, and dozens of other large enterprises (Help Net Security). The dumps include AWS secret access keys, Salesforce client secrets, Slack signing secrets, and Azure environment variables captured mid-pipeline. The uncomfortable part: any credential that was live in a CI runner during that 40-minute window is still valid until someone actively rotates it — removing the malicious package did nothing to invalidate what it already stole. If you had LiteLLM anywhere in a CI/CD dependency tree back in March, this is worth a credential audit today, not a \"we'll get to it\" item — check for AWS keys, Slack tokens, and cloud provider secrets that predate March 19 and haven't been rotated since.
AWS added support for configuring Kubernetes control plane parameters directly on EKS — scheduler behavior, controller-manager settings, and API server options that used to require self-managing the control plane to touch (AWS). The scheduler example AWS calls out is a good one: switching the node resource fit strategy from the default LeastAllocated (spread pods across nodes) to MostAllocated (pack nodes tightly) can meaningfully cut node count for workloads where you don't need the spare headroom. It's available in every region EKS runs in, no migration required. Worth a look if you've ever wanted more control over pod-placement or autoscaling responsiveness on EKS without giving up the managed control plane.
AWS extended the security-patch window for older Elasticsearch (1.5–7.8) and OpenSearch (1.0–1.2, 2.3–2.9) versions through November 7, 2027 — but from November 7, 2026, the Extended Support surcharge jumps to equal your instance cost, effectively doubling the price of running those old versions (AWS). If you've got domains sitting on a pre-7.9 Elasticsearch or early-2.x OpenSearch version because upgrading was never urgent, this is the forcing function: budget for either the upgrade or the doubled bill before November.
\nThe LiteLLM fallout is the one to act on today — a credential rotation check costs you an hour and closes a door that's been open since March. The AWS items are both \"plan now, act before the deadline\" — EKS control plane tuning whenever you get to it, OpenSearch versions before the November surcharge hits.
\n", "date_published": "2026-08-13T00:00:00.000Z", "tags": [ "roundup", "aws", "security" ] }, { "id": "https://jstgtech.com/blog/2026-08-12-cloud-roundup/", "url": "https://jstgtech.com/blog/2026-08-12-cloud-roundup/", "title": "Cloud roundup: Metabase CVSS 10 SQLi confirmed exploited", "summary": "A CVSS 10 Metabase SQLi is now confirmed exploited and in CISA KEV, plus an exploited Cisco ASA/FTD DoS bug and Microsoft's Patch Tuesday zero-day.", "content_html": "Another security-heavy day: the Metabase bug I flagged Monday as a zero-day with no CVE now has one and is confirmed under active exploitation, Cisco's VPN appliances picked up an exploited DoS flaw, and Microsoft's August Patch Tuesday landed with an already-exploited zero-day of its own.
\nCVE-2026-72898 is an unauthenticated SQL injection in Metabase's password-reset endpoint (POST /api/session/reset_password) that lets an attacker inject arbitrary SQL into the application database and walk away with admin access — no auth, no user interaction required, CVSS 10.0 (Bishop Fox). CISA added it to the Known Exploited Vulnerabilities catalog on August 11 (CISA). It affects a wide version range — 0.58 through 0.63.4 and the matching Enterprise 1.x builds — so if you're running self-hosted Metabase anywhere, including a \"just for internal dashboards\" instance, patch it now and don't wait for the next maintenance window. Admin takeover on a BI tool usually means access to every data source it's connected to, which for most of us is production databases and warehouses.
CVE-2026-20349 (CVSS 8.6) is a heap inspection bug in Cisco Secure Firewall ASA and FTD's Remote Access SSL VPN service — a crafted, unauthenticated HTTP request can crash the device and force a reload (The Hacker News). Cisco confirmed active exploitation and CISA added it to KEV in the same August 11 batch as the Metabase bug, with a federal remediation deadline of August 14. If you've got ASA/FTD devices with SSL VPN listeners facing the internet, this is a \"patch this week\" item — a device that keeps rebooting under attacker control is its own kind of outage, on top of whatever else the attacker is probing for while it's down.
\nMicrosoft's August Patch Tuesday shipped fixes for roughly 400 vulnerabilities including three zero-days, and confirmed one — CVE-2026-68820, a use-after-free elevation-of-privilege bug in the Windows Ancillary Function Driver for WinSock — is already being exploited in the wild (BleepingComputer). AFD.sys bugs are a recurring privilege-escalation path once an attacker has any foothold, so this is the one to prioritize on your Windows fleet — EC2 Windows instances, jump boxes, and anything else running Windows — even though the initial-access vector is elsewhere.
\nAmazon Connect added a performance dashboard for Cases, giving managers case-volume, resolution-trend, and SLA-attainment views without building custom reporting on top of the Cases API — a small but welcome addition if you're running support workflows through Connect.
\nTwo unauthenticated, actively-exploited bugs landed in KEV on the same day — Metabase SQLi and the Cisco ASA/FTD DoS — so triage internet-facing instances of either today. The Windows AFD zero-day is a \"patch it in the normal cycle, but don't push the normal cycle out\" item.
\n", "date_published": "2026-08-12T00:00:00.000Z", "tags": [ "roundup", "aws", "security" ] }, { "id": "https://jstgtech.com/blog/2026-08-11-cloud-roundup/", "url": "https://jstgtech.com/blog/2026-08-11-cloud-roundup/", "title": "Cloud roundup: New ransomware exploits N-central bug", "summary": "Microsoft ties new StormEncryptor ransomware to an N-able N-central auth bypass, plus new AWS EC2 health checks and DRS UEFI support for Linux failovers.", "content_html": "If you're still triaging the N-central saga from earlier this week, there's a new wrinkle: it's now ransomware. Otherwise it's a lighter AWS day — two operational upgrades worth bookmarking rather than dropping everything for.
\nMicrosoft disclosed that Storm-1175, a financially motivated China-linked actor that previously ran Medusa ransomware, has moved to a new strain called StormEncryptor — and the likely entry point is CVE-2026-18577, the N-able N-central authentication bypass CISA added to KEV earlier this week (The Hacker News). Microsoft assesses that CVE-2026-18577 is actually a patch bypass for the earlier CVE-2026-18556, meaning the first fix didn't fully close the door. Once in, Storm-1175 moves fast — AnyDesk or SimpleHelp for persistence, Advanced IP Scanner for discovery, Mimikatz for credential dumping, then exfiltration and encryption within days of initial access. If you're an MSP, or downstream of one running N-central, this is the concrete \"why\" behind all the patch-immediately warnings from earlier this week: confirm you're on the latest hotfix (not just the first one), and hunt for AnyDesk/SimpleHelp installs you didn't put there yourself.
\nAWS launched Application Status Checks for EC2 — you configure a protocol, port, path, and expected response code, and EC2 polls it every 60 seconds, feeding the result into the same status-check signal Auto Scaling already watches (AWS). It's live in all commercial regions plus GovCloud. The gap this closes: instance and system status checks tell you the VM is up, not that nginx crashed or the app inside stopped responding — that's traditionally been a custom health-check script or an ALB target group doing double duty. If you run EC2 outside an ALB, or want faster detection than an ALB's own health-check interval, this is a built-in way to get unhealthy instances replaced without writing your own watcher.
\nAWS Elastic Disaster Recovery now carries UEFI boot mode through to recovered Linux instances automatically, instead of dropping them into legacy BIOS mode and leaving you to fix boot config after a failover (AWS). No configuration change needed, no extra cost. It's a small thing until the day you actually fail over and discover your UEFI-dependent app won't boot — worth knowing about before you need it, not during an actual DR event.
\nAWS brought U7in-24TB high-memory EC2 instances to São Paulo for SAP HANA/Oracle-scale workloads, and bumped OpenSearch Serverless from 1,500 to 10,000 collections per collection group — useful if you're running dense multi-tenant search and were bumping into the old ceiling.
\nThe N-central story just got a lot more concrete: it's now actively being used to deploy ransomware, so if you haven't confirmed your hotfix level, do that today. On the AWS side, the two operational items — EC2 app-level health checks and DRS's UEFI fix — are both worth turning on now, quietly, before you need them.
\n", "date_published": "2026-08-11T00:00:00.000Z", "tags": [ "roundup", "aws", "security" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-cloud-roundup/", "url": "https://jstgtech.com/blog/2026-08-10-cloud-roundup/", "title": "Cloud roundup: LoadMaster RCE hits its KEV deadline today", "summary": "A critical unauthenticated Progress LoadMaster RCE hits its federal KEV remediation deadline today, plus a CVSS 10 Metabase zero-day and AWS supply chain security news.", "content_html": "If you run a Progress/Kemp LoadMaster load balancer, today's the deadline. Beyond that, it's a quieter day — another max-severity zero-day to know about, a bad update on last week's N-central saga, and AWS chipping away at supply chain risk.
\nCISA added CVE-2026-8037 (CVSS 9.6), an unauthenticated command injection flaw in Progress (Kemp) LoadMaster, to the KEV catalog on August 7 after eSentire and watchTowr Labs both reported active exploitation attempts — 792 reported attempts by one count (The Hacker News). The bug lives in a function called escape_quotes() and lets an unauthenticated attacker send unsanitized input to a LoadMaster API endpoint and get arbitrary command execution on the appliance. Progress shipped the fix back in June (GA 7.2.63.2 / LTSF 7.2.54.18), so this is really about finding the load balancers your org forgot to patch. Federal civilian agencies have until today, August 10, to remediate under BOD 26-04 — if you run LoadMaster anywhere with its management API exposed, that's your action item regardless of whether you're a federal shop.
Metabase — a popular open-source BI/dashboarding tool a lot of teams point at production databases — has a maximum-severity, unauthenticated SQL injection zero-day being actively exploited in the wild, including against Metabase Cloud itself (The Hacker News). An attacker can inject SQL through the app database with no credentials and land admin access, which means the credentials for every database Metabase is connected to are exposed too. It affects the 1.58 through 1.63 release lines; patched builds are out (1.58.24 through 1.63.5). If you can't patch immediately, block the /api/session/reset_password endpoint at your edge as a stopgap, then rotate the DB credentials Metabase held once you're patched — assume they were seen.
An update to the N-central story from earlier this week: N-able shipped Hotfix 2 after finding attackers who'd exploited CVE-2026-18577 were registering Cloudflare Tunnel connections on compromised managed endpoints to keep access even after the N-central server itself got locked down (The Hacker News). If you patched with Hotfix 1 and called it done, it's worth another look — N-able is now shipping expanded IOCs and a detection template, and explicitly warning that a clean scan doesn't guarantee you weren't hit. Good reminder that \"authentication bypass\" bugs on management platforms deserve a persistence hunt, not just a patch-and-move-on.
\nAWS added supply chain security as the tenth category in Security Hub Extended, with Chainguard and Socket as the curated partners, aimed at catching malicious code in open-source dependencies before it gets built into your app (AWS). It's pay-as-you-go with no commitment, and findings land in the same OCSF-normalized dashboard as everything else in Security Hub. If you're already paying for Extended and doing dependency scanning with a separate standalone tool, worth comparing — consolidating that signal into the same place as your other findings is generally a win for whoever's triaging.
\nPatch LoadMaster today if you run one exposed. If you're on Metabase, patch and rotate credentials — assume compromise until proven otherwise. And if you touched N-central last week, go back and hunt for persistence rather than trusting the first patch.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "roundup", "aws", "security" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-cicd-static-site-s3-cloudfront/", "url": "https://jstgtech.com/blog/2026-08-10-cicd-static-site-s3-cloudfront/", "title": "CI/CD for a static site: build, sync, and invalidate", "summary": "How to structure a GitHub Actions pipeline that builds a static site, syncs it to S3 with correct cache headers, and invalidates CloudFront without overpaying.", "content_html": "Deploying a static site sounds like it should be a solved problem: build it,\ncopy the files to a bucket, done. In practice the pipeline has three separate\nfailure modes that don't show up until production — stale HTML served from\ncache, orphaned objects left in the bucket after a rename, and an\ninvalidation bill that surprises you if you get cache-control wrong. This\nsite's own deploy pipeline (.github/workflows/deploy.yml) hits all three\nconcerns in about 20 lines, so I'll use it as the working example rather than\na hypothetical.
A static-site deploy is really three distinct jobs glued together:
\ndist/ directory of static files.They have to run in that order and each has to fully succeed before the next\nstarts — a partial sync followed by an invalidation just serves a broken mix\nof old and new files faster. Keep them as separate steps (not one giant\nshell script) so a failure at step 2 shows up clearly in the Actions log\ninstead of buried in a 40-line run: block.
- run: npm run build\n\n- name: Sync to S3\n run: aws s3 sync ./dist \"s3://my-bucket\" --delete\n\n- name: Invalidate CloudFront\n run: |\n aws cloudfront create-invalidation \\\n --distribution-id \"$DISTRIBUTION_ID\" \\\n --paths \"/*\"\nThat's the naive version. It works, but it's wrong in two specific ways —\nneither obvious until you look closely at cache-control and --delete\ntogether.
aws s3 sync doesn't set a useful Cache-Control header by default — S3\nserves objects with no explicit caching directive, which browsers and\nCloudFront interpret conservatively. For a static site built by a bundler\n(Astro, Vite, Next static export, etc.), you want two very different caching\npolicies in the same deploy:
/_astro/chunk-a1b2c3.js, hashed CSS, images) —\nthe filename changes when the content changes, so it's safe to cache these\nforever. public, max-age=31536000, immutable./blog/index.html) but\nthe content changes on every deploy. These need max-age=0,\n must-revalidate so a browser or CDN edge always re-checks before serving\na cached copy.You can't set one blanket --cache-control flag for the whole sync and get\nboth right, so split it into two sync calls with complementary\n--exclude/--include filters:
- name: Sync to S3\n run: |\n aws s3 sync ./dist \"s3://$BUCKET\" \\\n --delete \\\n --exclude \"*.html\" \\\n --exclude \"*.xml\" \\\n --cache-control \"public,max-age=31536000,immutable\"\n\n aws s3 sync ./dist \"s3://$BUCKET\" \\\n --delete \\\n --exclude \"*\" \\\n --include \"*.html\" \\\n --include \"*.xml\" \\\n --cache-control \"public,max-age=0,must-revalidate\"\nThis is exactly what this site's deploy.yml does. The --exclude/\n--include pairs are inverses of each other on purpose — every object in\ndist/ is claimed by exactly one of the two calls, never both, never\nneither.
--delete gotcha: it respects filters, both waysHere's the part that isn't obvious from the CLI docs' one-line description.\naws s3 sync --delete removes destination objects that aren't present in\nthe source and match the command's own filters. It does not delete\neverything in the bucket that the source lacks — it only considers objects\nwithin the scope defined by --exclude/--include for that specific\ninvocation.
That's exactly why running the two-pass sync above is safe: the first call\n(non-HTML) only ever deletes stale non-HTML objects, and the second call\n(HTML/XML) only ever deletes stale HTML/XML objects. Neither pass can\naccidentally delete the other's files, because each treats them as excluded\nand therefore invisible.
\nThe gotcha is what happens if your two filter sets aren't exact\ncomplements — say you add a new file extension to the build output (a\n.webmanifest, a .txt) and forget to add it to either pass's include\nlist. It won't get deleted when removed (each --delete ignores it), but it\nalso won't get the cache-control header you intended on either pass — it\nsilently falls through both filters. Test a rename/removal locally against a\nscratch bucket (aws s3 sync --dryrun) whenever you touch the exclude\npatterns, not just when you add new ones.
The instinct once you've fixed caching is to invalidate narrowly — pass the\nexact paths that changed instead of /*, on the theory that a full\nwildcard invalidation is expensive because it touches every object at the\nedge. That instinct is backwards for CloudFront specifically, and it's worth\nknowing why before you build a diff-based \"only invalidate what changed\"\nstep.
CloudFront invalidation pricing is **per path string submitted in the\nrequest, not per object matched**. The first 1,000 paths per month are\nfree; after that it's $0.005 per path. --paths \"/*\" is one path as far\nas billing is concerned, regardless of how many thousands of objects it\nactually clears. Compare that to a \"smart\" pipeline that diffs the build and\nsubmits one path per changed file — a typical content update touching 15\nfiles costs the same order of magnitude as 15 separate wildcard deploys, and\nif you ever invalidate per-object on a big rebuild (hundreds of pages) you\ncan burn through the free tier in a single deploy.
For a low-traffic personal site or portfolio, /* on every deploy is both\nsimpler and, counter-intuitively, usually cheaper than trying to be\nclever about scoping. It's the right default. The one place scoping earns\nits complexity is a high-frequency deploy pipeline (many deploys per hour,\ne.g. a CMS with instant-publish) where wildcard invalidations would\notherwise queue up and a --paths \"/blog/*\" \"/index.html\" pattern targeting\nonly the collections that actually changed keeps the queue from backing up —\nCloudFront processes invalidations from a single distribution somewhat\nserially, and a backlog of full-site wildcards delays the one that matters.
- name: Invalidate CloudFront\n run: |\n aws cloudfront create-invalidation \\\n --distribution-id \"$DISTRIBUTION_ID\" \\\n --paths \"/*\"\nIf you do scope it, don't hand-roll the diff from git diff --name-only —\nmap source paths to routes, not files. A change to a shared layout\ncomponent invalidates every page that uses it, not the one file that\nchanged; a naive file-based diff will under-invalidate and leave stale pages\nin cache with no error to tell you it happened.
Add the two-pass sync and cache-control split first, deploy once, and check\nresponse headers with curl -I against a fingerprinted asset and against\n/ to confirm the immutable/must-revalidate split landed correctly before\nyou touch the invalidation step. Then switch the invalidation from whatever\nad-hoc scoping you had to /* and watch a billing cycle — for most personal\nand small-business traffic levels you'll stay inside the free 1,000-path\ntier for months. Only reach for path-scoped invalidations once you have\nactual evidence (a CloudFront invalidation queue backing up, or genuinely\nexceeding the free tier) rather than optimizing against an assumption about\ncost that, for this specific service, runs the opposite direction from most\npeople's intuition.
If your deploy pipeline ends every push with `aws cloudfront create-invalidation\n--paths \"/*\"`, it works, but it's the CDN equivalent of clearing your entire\nbrowser cache because one tab was stale. It's slow to propagate, it costs\nmoney past a fairly small free tier, and — the part people miss — it doesn't\nactually need to exist for most of your assets if you name your files right.\nThis site's own deploy (deploy.yml) does a scoped invalidation of exactly\ntwo paths, not a wildcard, and here's the reasoning and the mechanics behind\nthat choice.
CloudFront gives you **1,000 free invalidation path requests per month, per\naccount**. Past that, each additional path costs $0.005. The part that trips\npeople up is what counts as \"one path\":
\n/* — a single wildcard path — counts as one path, no matter how many\nobjects it matches at the edge. Nuking your entire distribution with one\nwildcard call costs the same as invalidating one specific file.create-invalidation call is billed\nper path in the list. aws cloudfront create-invalidation --paths\n \"/index.html\" \"/blog/index.html\" \"/rss.xml\" bills three paths, not one call.So a single /* is actually the cheapest way to invalidate by path-count —\nthe trap isn't cost from the wildcard itself, it's what a wildcard does\noperationally: it forces CloudFront to revalidate every object in the\ndistribution against the origin on the next request, which means a burst of\norigin requests to S3 right after every deploy, and it invalidates objects\nthat never changed. On a low-traffic personal site that's harmless. On\nanything with real traffic or a slow/rate-limited origin, a full-distribution\nwildcard after every deploy is the thing that actually costs you — in origin\nload and in cache-miss latency for visitors who hit the edge in the seconds\nafter the invalidation lands, not in the $0.005-per-path line item.
The free tier resets monthly and is shared across all distributions in the\naccount, so if you run several sites or a multi-tenant setup off one AWS\naccount, a chatty pipeline on one distribution eats the free allowance for\nall of them.
\nInvalidation is a workaround for a caching mistake: telling CloudFront to\ncache a URL for a long time when the content behind that URL can change.\nFingerprinted (content-hashed) filenames remove the mistake instead of\ncompensating for it. Astro's build already does this for you — run npm run\nbuild and look at dist/_astro/:
_astro/client.a1b2c3d4.js\n_astro/index.e5f6a7b8.css\nThe hash is derived from the file's content. Change one character of source,\nthe hash changes, the URL changes. Because the URL is different, there's\nnothing to invalidate — the old URL still exists at the edge serving old\ncontent (fine, nothing references it anymore), and the new URL is a cache\nmiss exactly once, everywhere, the first time each edge location requests it.\nThis is why you can safely set:
\nCache-Control: public, max-age=31536000, immutable\non everything under _astro/ (or /assets/, however your bundler names it).\nA year-long max-age plus immutable tells both browsers and CloudFront\n\"never revalidate this, ever\" — and that's true, because the filename itself\nguarantees the content can't change out from under that URL. This is the\nsingle biggest lever for cutting invalidation traffic to near zero: the\nmajority of a static site's bytes (JS, CSS, hashed images) never need an\ninvalidation call in their entire lifetime.
Set this at the S3 origin via object metadata (Astro/most bundlers don't set\nCache-Control on upload themselves — your sync step has to), or override it\nat the CloudFront cache behavior level with a policy scoped to the hashed\nasset path pattern:
aws s3 sync ./dist s3://my-bucket/ \\\n --exclude \"*\" --include \"_astro/*\" \\\n --cache-control \"public, max-age=31536000, immutable\" \\\n --metadata-directive REPLACE\nRun this pass before the pass that uploads everything else, so the\nhashed-assets rule doesn't get clobbered by a broader default Cache-Control\napplied later in the sync.
Fingerprinting only works for files whose name changes when their content\nchanges. Two categories of file don't get that treatment, and those are the\nonly things worth invalidating:
\nindex.html and any other non-hashed HTML entrypoint. The URL\n/blog/index.html (or /blog/ via the pretty-URL rewrite) has to stay\nstable — it's what's in every bookmark, backlink, and search index entry —\nbut its content changes every time you publish. Give these a short\nmax-age (or no-cache so CloudFront/browsers always revalidate against\norigin) and invalidate them explicitly on deploy.favicon.ico,\nrobots.txt, sitemap-index.xml, or rss.xml — anything a spec or a\nclient expects at a fixed path.For this site, that means the deploy step invalidates a short, explicit list,\nnot a wildcard:
\naws cloudfront create-invalidation \\\n --distribution-id \"$CF_DISTRIBUTION_ID\" \\\n --paths \"/index.html\" \"/blog/*\" \"/rss.xml\" \"/sitemap-index.xml\"\n/blog/* is doing real work here — every post and tag page under /blog/\nis HTML with a stable, non-hashed URL, so a scoped wildcard on just that\nsubtree is the right call. It's still one billed path (wildcards always are),\nbut more importantly it only forces revalidation on the part of the site that\nactually changes on every deploy, leaving the immutable hashed assets alone.
create-invalidation returns immediately with a status of InProgress — the\nCLI call succeeding does not mean the content is gone from every edge\nlocation yet. Full propagation across all of CloudFront's edge locations\ntypically completes within a few minutes, but there's no SLA guaranteeing a\nspecific time, and it's not uncommon to see stale content served from one\nedge location after another has already updated. If your deploy pipeline\nruns a post-deploy smoke test that curls the live URL immediately after\ncreate-invalidation returns, don't assert on content freshness right away —\npoll get-invalidation for Status: Completed first, or just accept some\npropagation lag in the check:
aws cloudfront get-invalidation \\\n --distribution-id \"$CF_DISTRIBUTION_ID\" \\\n --id \"$INVALIDATION_ID\" \\\n --query 'Invalidation.Status'\nThe second, less obvious gotcha: **invalidation requests queue per\ndistribution**, and each request can list at most 3,000 paths (or 15 with\nwildcards). If a script fires invalidations in a tight loop — say, a bulk\ncontent migration that invalidates per-file instead of batching — later\nrequests sit InProgress behind earlier ones rather than running in\nparallel, so a burst of small invalidations can take noticeably longer to\nfully clear than one batched call with the same total path count. Batch your\npaths into as few create-invalidation calls as the file-list limits allow,\nrather than looping a call per file.
Fingerprint everything your bundler can fingerprint and set max-age=31536000,\nimmutable on it — that's most of your bytes and it needs zero invalidation\ncalls, ever, for the life of the file. For the small set of non-hashed\nentrypoints (HTML, robots.txt, feeds), use a short max-age and a scoped\nexplicit-path invalidation on deploy, not /*. You'll stay comfortably\ninside the free 1,000-path monthly allowance even with several deploys a day,\nand — the bigger win — your origin only gets hit for the handful of objects\nthat actually changed, not your entire distribution.
If your GitHub Actions workflows still authenticate to AWS with a stored\nAWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY pair, you're carrying a\nlong-lived credential that can leak from a log, a fork's pull_request_target\nrun, or a compromised dependency — and it keeps working until someone\nremembers to rotate it. OpenID Connect (OIDC) federation gets rid of that\nentirely: GitHub mints a short-lived, workflow-scoped identity token, AWS STS\ntrades it for temporary credentials, and there's nothing sitting in your repo\nsecrets for an attacker to steal. This is the same pattern I use for this\nsite's own deploy pipeline, so what follows is the working setup, not just\nthe AWS docs paraphrased.
https://token.actions.githubusercontent.com. Every workflow run can\nrequest a signed JWT from it (via id-token: write permission) with\nclaims describing the repo, branch, and workflow.sub claim matches this specific repo and\nref.\"aws-actions/configure-aws-credentials action exchanges the JWT for\ntemporary STS credentials scoped to that role, valid for the run only.No secret ever leaves GitHub's control plane. Nothing to rotate, nothing to\nrevoke except the role's trust policy.
\nDo this once per AWS account (Terraform, since you're presumably managing\nthe rest of your IAM this way too):
\nresource \"aws_iam_openid_connect_provider\" \"github_actions\" {\n url = \"https://token.actions.githubusercontent.com\"\n client_id_list = [\"sts.amazonaws.com\"]\n thumbprint_list = [\"6938fd4d98bab03faadb97b34396831e3780aea1\"]\n}\nThat thumbprint is GitHub's OIDC endpoint CA thumbprint. Gotcha: AWS\nactually ignores this field for GitHub's provider now (it validates via the\nstandard TLS CA bundle instead), but the argument is still required by the\nresource — don't spend time trying to keep it \"current,\" it's a legacy\nrequirement AWS kept for backward compatibility.
\nThis is where most setups go wrong. The sub claim format is\nrepo:<org>/<repo>:<qualifier>, and it's tempting to wildcard it into\nuselessness:
{\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"Federated\": \"arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com\"\n },\n \"Action\": \"sts:AssumeRoleWithWebIdentity\",\n \"Condition\": {\n \"StringEquals\": {\n \"token.actions.githubusercontent.com:aud\": \"sts.amazonaws.com\"\n },\n \"StringLike\": {\n \"token.actions.githubusercontent.com:sub\": \"repo:my-org/jstgtech-web:ref:refs/heads/main\"\n }\n }\n}\nTwo things to get right:
\naud condition. Without it, any GitHub Actions run\nanywhere that requests a token for sts.amazonaws.com audience and\nhappens to match your sub pattern can assume the role. aud is cheap\ninsurance and AWS's own quickstart includes it — don't skip it because the\nconsole wizard makes it feel optional.sub to the exact ref, not just the repo. repo:my-org/my-repo:*\nlets a PR from a fork-turned-branch, a tag push, or an environment:\ndeployment all assume the same role your production deploy uses. If a\nworkflow only needs to deploy from main, pin sub to\nrepo:my-org/jstgtech-web:ref:refs/heads/main. If you use GitHub\nEnvironments for a manual-approval gate, scope to\nrepo:my-org/jstgtech-web:environment:production instead — that ties the\nAWS role to the same approval gate protecting your environment secrets.For a PR-only workflow (say, terraform plan on pull requests, no write\naccess), use a separate, more restrictive role with sub matching\nrepo:my-org/jstgtech-web:pull_request and a read-only policy attached —\ndon't reuse your deploy role's trust policy with a broader condition \"just\nfor now.\"
permissions:\n id-token: write\n contents: read\n\njobs:\n deploy:\n runs-on: ubuntu-latest\n steps:\n - uses: aws-actions/configure-aws-credentials@v4\n with:\n role-to-assume: arn:aws:iam::123456789012:role/github-actions-site-deploy\n aws-region: us-east-1\npermissions: id-token: write is not optional — it's what lets the runner\nrequest the JWT in the first place, and GitHub defaults this to none at\nthe org or repo level on newer accounts. If you get `Error: Not authorized\nto perform sts:AssumeRoleWithWebIdentity`, check this before anything else;\nit's the single most common cause, ahead of trust-policy typos.
MaxSessionDuration still\ngoverns how long the assumed credentials last. Keep it at the default\n(1 hour) or lower for deploy roles — there's no reason a CI job needs an\n8-hour session.token.actions.githubusercontent.com (GitHub issues it, not the\nrunner), so this setup doesn't need to change — but it's worth confirming\nif you're debugging a runner migration and OIDC suddenly stops working for\nan unrelated reason (usually a network path to GitHub's OIDC endpoint from\nthe runner).AdministratorAccess is still one compromised Action away from\na bad day — scope the permissions policy to exactly the S3 bucket,\nCloudFront distribution, or Terraform state path the workflow touches.If you're migrating an existing pipeline off access keys, run both in\nparallel for one deploy cycle: add the OIDC role, switch\nconfigure-aws-credentials to role-to-assume, watch a real deploy\nsucceed, then delete the IAM user and its access keys. Don't delete the\nold credentials in the same PR that introduces the new role — if the trust\npolicy's sub condition is wrong, you want a fallback for the next deploy\ninstead of a broken pipeline and no way back in until you fix IAM by hand.
If you're running more than one or two AWS accounts, you've probably got the\nsame mess I inherited: a pile of IAM users, one per human, duplicated in\nevery account they need access to, each with its own password, its own MFA\ndevice, and — if you're unlucky — its own long-lived access keys sitting in\nsomeone's ~/.aws/credentials. Nobody remembers to deprovision the ex-\ncontractor's user in the staging account. Nobody's rotated the access keys\non the prod-readonly user since 2023. IAM Identity Center (the renamed AWS\nSSO service) fixes this by moving identity out of individual accounts\nentirely: you log in once, in one place, and get temporary credentials into\nwhichever accounts and roles you're assigned to. This is the setup I run\nacross a management account plus workload accounts, with the actual\nTerraform and the parts that don't work the way the docs imply.
Per-account IAM users have three structural problems: identity is\nduplicated per account (N users × M accounts to manage), credentials are\nusually long-lived (passwords, access keys), and there's no single place to\nsee \"what can this person get into.\" IAM Identity Center centralizes all of\nthat:
\nThe end state: a human logs into the Identity Center portal URL once,\nsees a tile for every account/role combination they've been granted, clicks\none, and gets temporary STS credentials. No IAM user, no password in that\naccount, no access key to leak.
\nIdentity Center is enabled once per AWS Organization, in one region (it's\nfree-tier, but the instance and its permission sets live in a single home\nregion even though assignments apply org-wide). I run mine out of\nus-east-1 in the management account, alongside the org's other org-wide\nresources.
If you already run Okta or Entra ID, use it — don't stand up a second\nidentity directory to maintain. Enable SCIM provisioning in Identity Center\n(Settings → Identity source → Automatic provisioning), which gives you an\nSCIM endpoint URL and a bearer token. Paste those into your IdP's SCIM app\nconfig, and your IdP becomes the source of truth for users and groups;\nIdentity Center just mirrors them.
\nIf you don't have an external IdP yet, the built-in Identity Center\ndirectory is fine to start with — you can migrate to an external IdP later\nwithout re-doing your permission sets or assignments, since those reference\ngroup IDs, not the identity source itself.
\nA permission set is either a collection of AWS managed policies, a custom\ninline policy, or both, plus a session duration. I manage mine in Terraform\nusing the aws_ssoadmin_permission_set resource — this is the piece people\nskip because the console makes it feel like a one-off, and then six months\nlater nobody remembers what's in the Billing-ReadOnly permission set or\nwhy.
data \"aws_ssoadmin_instances\" \"this\" {}\n\nresource \"aws_ssoadmin_permission_set\" \"read_only\" {\n name = \"ReadOnlyAccess\"\n instance_arn = tolist(data.aws_ssoadmin_instances.this.arns)[0]\n session_duration = \"PT4H\"\n}\n\nresource \"aws_ssoadmin_managed_policy_attachment\" \"read_only\" {\n instance_arn = tolist(data.aws_ssoadmin_instances.this.arns)[0]\n permission_set_arn = aws_ssoadmin_permission_set.read_only.arn\n managed_policy_arn = \"arn:aws:iam::aws:policy/ReadOnlyAccess\"\n}\n\nresource \"aws_ssoadmin_permission_set\" \"billing_admin\" {\n name = \"BillingAdmin\"\n instance_arn = tolist(data.aws_ssoadmin_instances.this.arns)[0]\n session_duration = \"PT1H\"\n}\n\nresource \"aws_ssoadmin_permission_set_inline_policy\" \"billing_admin\" {\n instance_arn = tolist(data.aws_ssoadmin_instances.this.arns)[0]\n permission_set_arn = aws_ssoadmin_permission_set.billing_admin.arn\n inline_policy = jsonencode({\n Version = \"2012-10-17\"\n Statement = [{\n Effect = \"Allow\"\n Action = [\"aws-portal:*Billing\", \"aws-portal:*Usage\", \"ce:*\"]\n Resource = \"*\"\n }]\n })\n}\nsession_duration is an ISO 8601 duration, and it's the actual cap on\ncredential lifetime for anyone assigned this permission set — not\noverridable at assumption time. I keep BillingAdmin at PT1H and\nReadOnlyAccess at PT4H since read-only browsing sessions are lower risk\nthan anything that touches spend controls.
This is the part that trips people up coming from single-account IAM\nthinking: a permission set is a template, not a role. Nothing exists in a\nmember account until you create an account assignment — at that point\nIdentity Center provisions an actual IAM role in the target account, named\nAWSReservedSSO_<permission-set-name>_<hash>, with a trust policy that\nallows the Identity Center service to assume it. You'll see these roles\nshow up in IAM → Roles in every account you've assigned; don't hand-edit\nthem, they're managed by Identity Center and your edits will get silently\nreverted on the next provisioning sync.
Assignment is a three-way link — group, permission set, account:
\nresource \"aws_ssoadmin_account_assignment\" \"readonly_staging\" {\n instance_arn = tolist(data.aws_ssoadmin_instances.this.arns)[0]\n permission_set_arn = aws_ssoadmin_permission_set.read_only.arn\n\n principal_id = data.aws_identitystore_group.engineers.group_id\n principal_type = \"GROUP\"\n\n target_id = \"222233334444\" # staging account\n target_type = \"AWS_ACCOUNT\"\n}\nSay I want the Engineers group to have ReadOnlyAccess in dev,\nstaging, and prod, but a separate PowerUser permission set only in\ndev. First, look up the group by its SCIM-synced display name:
data \"aws_identitystore_group\" \"engineers\" {\n identity_store_id = tolist(data.aws_ssoadmin_instances.this.identity_store_ids)[0]\n\n alternate_identifier {\n unique_attribute {\n attribute_path = \"DisplayName\"\n attribute_value = \"Engineers\"\n }\n }\n}\n\nlocals {\n readonly_accounts = {\n dev = \"111122223333\"\n staging = \"222233334444\"\n prod = \"333344445555\"\n }\n}\n\nresource \"aws_ssoadmin_account_assignment\" \"readonly\" {\n for_each = local.readonly_accounts\n\n instance_arn = tolist(data.aws_ssoadmin_instances.this.arns)[0]\n permission_set_arn = aws_ssoadmin_permission_set.read_only.arn\n principal_id = data.aws_identitystore_group.engineers.group_id\n principal_type = \"GROUP\"\n target_id = each.value\n target_type = \"AWS_ACCOUNT\"\n}\n\nresource \"aws_ssoadmin_account_assignment\" \"poweruser_dev\" {\n instance_arn = tolist(data.aws_ssoadmin_instances.this.arns)[0]\n permission_set_arn = aws_ssoadmin_permission_set.power_user.arn\n principal_id = data.aws_identitystore_group.engineers.group_id\n principal_type = \"GROUP\"\n target_id = local.readonly_accounts.dev\n target_type = \"AWS_ACCOUNT\"\n}\nApply that and everyone in Engineers sees three tiles for ReadOnlyAccess\n(one per account) and one extra tile for PowerUser in dev, in the\nIdentity Center portal. If you'd rather click through the console: Identity\nCenter → Permission sets → select one → nothing happens there for\nassignment; you actually go to Multi-account permissions → select\naccount(s) → select the permission set → select the group. It's easy to\nland on the wrong page first time because assignment lives under the\naccount list, not the permission set page.
aws sso loginOnce assigned, configure a named profile pointing at the SSO session:
\n# ~/.aws/config\n[sso-session my-org]\nsso_start_url = https://my-org.awsapps.com/start\nsso_region = us-east-1\nsso_registration_scopes = sso:account:access\n\n[profile staging-readonly]\nsso_session = my-org\nsso_account_id = 222233334444\nsso_role_name = ReadOnlyAccess\nregion = us-east-1\naws sso login --profile staging-readonly\naws sts get-caller-identity --profile staging-readonly\naws sso login opens a browser, you authenticate against your IdP (or the\nbuilt-in directory), and the CLI caches temporary credentials locally,\nscoped to the permission set's session_duration. When they expire, you\njust re-run aws sso login — no key rotation, nothing to revoke in the\naccount itself, because there was never a long-lived credential there to\nbegin with.
aws sso-admin provision-permission-set triggered automatically by\nterraform apply for aws_ssoadmin_permission_set changes — but if\nyou're editing an inline policy attached via a separate resource, double\ncheck the apply actually touched the permission set resource itself, not\njust the attachment, or the role in the target account can silently stay\nstale for hours.aws_identitystore_group data source in Terraform can't find it,\nthat's not a Terraform bug, it's sync lag. Check Identity Center →\nSettings → Automatic provisioning for the last sync timestamp before you\nstart debugging your HCL.aws_ssoadmin_account_assignment state entries hang around\nif the group was deleted out-of-band from Terraform. terraform plan\nwill complain it can't find the referenced group ID. Clean up\nassignments in code before deleting the group upstream, not after.AssumeRole into with a custom DurationSeconds up to the role's\nmax, the permission set's session_duration is what you get — there's no\nper-login override. If your CI needs longer sessions than your interactive\npermission set allows, that's a sign CI shouldn't be using an Identity\nCenter human-login flow at all; use a workload identity (OIDC role\nassumption, like I covered in the GitHub Actions post) instead.AdministratorAccess in the management account because it's first in the\naccount picker.aws sts get-caller-identity and\nCloudTrail both show the assumed-role session name, which Identity Center\npopulates with the actual user's identity — so AssumedRole events in\nCloudTrail are attributable to a person, not a shared IAM user ARN that\nthree people know the password to.aws sso login are for humans. Machine-to-machine\nauth (CI/CD, Lambda, EC2) should stay on IAM roles with OIDC federation\nor instance profiles — don't try to shoehorn a service account into an\nIdentity Center permission set just to avoid having two auth patterns in\nyour org.Don't flip every account to Identity Center-only in one PR. Stand up\nIdentity Center, get SCIM sync working, create permission sets, and assign\nthem in one low-risk account (I used a scratch sandbox account) first —\nconfirm people can actually log in via the portal and get the access you\nexpect. Then extend account assignments outward account by account, and\nonly after everyone's confirmed they can get in through SSO do you go back\nand deprovision the old per-account IAM users and their access keys. Keep a\nshort overlap window rather than a hard cutover date; the failure mode of\n\"we deleted the IAM user before confirming the SSO group assignment\nactually worked\" is a locked-out engineer and an emergency root login, not\na fun afternoon.
\"Just use provisioned concurrency\" is the answer I hear most often when\nsomeone complains about Lambda cold starts, and it's usually wrong — it fixes\nthe symptom, costs money every hour whether or not you're invoked, and skips\nthe part where you find out why your function is slow to initialize in the\nfirst place. Most cold start problems are fixable for free: a leaner\ndeployment package, a runtime that doesn't need to JIT-warm, or removing a VPC\nattachment that was never actually necessary. This is the diagnostic process I\nuse before reaching for the checkbook.
\nEvery Lambda invocation writes a REPORT line to CloudWatch Logs. On a cold\nstart, it includes an Init Duration field that the warm-start version\ndoesn't have:
REPORT RequestId: 8f3e... Duration: 412.33 ms Billed Duration: 413 ms\nMemory Size: 512 MB Max Memory Used: 98 MB Init Duration: 621.47 ms\nThat Init Duration is the number to chase — it's time spent on the\nexecution environment bootstrapping the runtime, running module-level code\noutside your handler, and (if configured) resolving your VPC ENI. It is not\nincluded in billed duration for most runtimes, so it won't show up as a cost\nspike, only as added latency your caller feels. Don't confuse it with\nDuration, which is your handler's own execution time and is what you're\nbilled for.
To pull this across many invocations instead of eyeballing one log line, use\nCloudWatch Logs Insights:
\nfields @timestamp, @initDuration, @duration\n| filter ispresent(@initDuration)\n| stats count(*) as coldStarts,\n avg(@initDuration) as avgInit,\n pct(@initDuration, 95) as p95Init\n by bin(1h)\nThe filter ispresent(@initDuration) line is what isolates cold starts —\nwarm invocations don't emit that field at all, so this query effectively\ngives you a cold-start rate and latency distribution per hour, for free, with\nno X-Ray required.
X-Ray is worth turning on (Tracing: Active in the function config) when you\nneed to see where inside init time is going — module imports, SDK client\nconstruction, secrets/config fetched at startup — rather than just how long\nit took. In the trace timeline, the Initialization segment sits before your\nfirst subsegment; if it's dominated by something like a boto3 client build\nor a config file fetched from Secrets Manager at import time, that's your\ntarget, not the runtime itself.
Interpreted runtimes (Python, Node.js, Ruby) have low inherent init\noverhead — a few hundred milliseconds — because there's no compilation step,\njust interpreter startup and module loading. Compiled/JIT runtimes (Java, C#\non .NET, and to a lesser extent Go, which compiles to a static binary ahead of\ntime) trade that off: Go's cold start is often the fastest of all because\nthere's no runtime to boot at all, while JVM- and CLR-based functions pay for\nclass loading and JIT warm-up on every cold start, frequently 1-3 seconds for\nanything beyond a trivial handler.
\nIf you're stuck on Java for ecosystem reasons, Lambda SnapStart is the\nsingle biggest lever available: it takes a pre-initialized, encrypted\nsnapshot of your execution environment's memory and disk state after your\nstatic initializers and any registered beforeCheckpoint hooks run, then\nresumes from that snapshot on cold start instead of re-running init from\nscratch. In practice this takes Java functions from multi-second cold starts\ndown to sub-200ms for many workloads. It's opt-in per function\n(SnapStart: ApplyOn: PublishedVersions), only applies to published\nversions (not $LATEST), and anything non-deterministic in your static\ninit — random values, UUIDs, timestamps, opened network connections — needs\nto be regenerated in a beforeCheckpoint/afterRestore hook, or you'll ship\nthe same \"random\" value to every restored environment. That's the gotcha that\nbites people first: a cached DB connection captured in the snapshot resumes\nin a stale, sometimes already-closed state on the other side.
Init duration scales with how much code Lambda has to unzip and load before\nyour handler is reachable, and it's rarely your own code that's the problem —\nit's the dependency tree. A Node function that pulls in the entire AWS SDK v2\n(aws-sdk) when it calls one S3 method drags in tens of megabytes it never\ntouches at runtime.
# See what's actually contributing to package size\ndu -sh node_modules/* | sort -rh | head -10\n\n# Node: import only the client you use (SDK v3 is modular by design)\nnpm uninstall aws-sdk\nnpm install @aws-sdk/client-s3\nFor Python, the equivalent is trimming requirements.txt to what's imported\nat module scope and pushing anything only used inside rarely-hit code paths\nto a lazy import inside the function body — module-level imports run during\ninit, so a heavy library imported \"just in case\" costs every cold start, not\njust the invocations that use it. For any runtime, moving large,\nrarely-changing dependencies into a Lambda Layer doesn't reduce\nunzip-and-load time by itself, but it does let you avoid re-uploading (and\nLambda re-validating) a multi-hundred-MB deployment package on every code\nchange, which matters more for deploy latency than cold starts.
Container-image Lambdas deserve a specific warning here: they're pulled from\nECR and have historically had noticeably worse cold starts than zip packages\nat larger image sizes, though Lambda's own image-caching layer has narrowed\nthat gap significantly since launch. If you're on container images purely out\nof habit and your image is small, a zip package with a layer is very likely\nfaster to cold-start.
\nBefore 2019, attaching a Lambda to a VPC meant provisioning an ENI per\nconcurrent execution environment, which could add 10 seconds or more to a\ncold start. AWS's Hyperplane-based networking model eliminated most of that\nby sharing ENIs across functions in the same VPC/subnet/security-group\ncombination, and current VPC-attached cold starts are typically within\ntens to a couple hundred milliseconds of non-VPC ones. If you're still\ncarrying a workaround from that era — a warm-up cron job, an oversized\nprovisioned-concurrency pool sized for the old ENI cost — it's worth\nre-measuring with the Logs Insights query above before assuming you still\nneed it. The overhead that remains is small but non-zero, so don't attach\na function to a VPC it doesn't need just because a sibling function does;\nscope VPC config per function, not per stack.
\nProvisioned concurrency pre-initializes a pool of execution environments and\nkeeps them warm, billed hourly whether invoked or not — it doesn't reduce\ninit duration, it just makes sure fewer invocations ever hit it. It's worth\nthe always-on cost when:
\nIt's overkill for async, batch, or event-driven functions (S3/SQS/EventBridge\ntriggers) where an extra few hundred milliseconds on an occasional invocation\nis invisible to anyone. Application Auto Scaling can scale provisioned\nconcurrency on a schedule (business-hours-only) or target-tracking policy, so\nif you do need it, don't just set a flat number sized for peak — that's\npaying peak-capacity prices around the clock for a curve that isn't flat.
\nStart with the Logs Insights query above running against production traffic\nfor a week to get a real cold-start rate and p95 Init Duration baseline —\ndon't optimize against a guess. Fix package bloat and runtime choice first;\nboth are free and often cut init duration by half or more on their own. Only\nreach for provisioned concurrency once you've re-measured after those changes\nand still have a latency-sensitive path that misses its SLA — and even then,\nsize it with Application Auto Scaling against your real traffic curve, not a\nnumber that felt safe.
Most IAM policies I inherit from a team are scoped by resource ARN and\nnothing else: \"this role can ec2:* on these instances.\" That's a start,\nbut it leaves gaps a determined or careless caller can walk through — an\nengineer's laptop credentials reaching production from a coffee shop, a\nrole that can touch every EC2 instance in the account regardless of which\nteam owns it, a sensitive action that only checks \"are you authenticated\"\nrather than \"did you actually use MFA to get here.\" IAM's condition keys\nclose those gaps without adding a second system to manage. They live right\ninside the policy document, they're evaluated by the same engine, and once\nyou know the handful that matter, they cover almost every real-world\nscoping requirement. Here's how I use them in practice, on real production\npolicies.
If your account has more than one team's resources in it, tag-based\nscoping is the highest-leverage condition you can add. Instead of\nenumerating ARNs (which breaks the moment someone launches a new\ninstance), you scope by tag and let your tagging discipline do the work:
\n{\n \"Effect\": \"Allow\",\n \"Action\": [\"ec2:StartInstances\", \"ec2:StopInstances\", \"ec2:RebootInstances\"],\n \"Resource\": \"arn:aws:ec2:*:123456789012:instance/*\",\n \"Condition\": {\n \"StringEquals\": {\n \"aws:ResourceTag/team\": \"platform\"\n }\n }\n}\naws:ResourceTag checks the tag already on the resource being acted on —\nuse it for read/modify/delete actions. For actions that create a\nresource, there's no tag on it yet at evaluation time, so you need\naws:RequestTag instead, checking the tag the caller is trying to apply:
{\n \"Effect\": \"Allow\",\n \"Action\": \"ec2:RunInstances\",\n \"Resource\": \"arn:aws:ec2:*:123456789012:instance/*\",\n \"Condition\": {\n \"StringEquals\": {\n \"aws:RequestTag/team\": \"platform\"\n }\n }\n}\nGotcha: this only stops people who use the console or API correctly —\nit does nothing if the tag can simply be omitted or changed later. Pair it\nwith a Deny that blocks untagged creation and, separately, blocks\nec2:DeleteTags / ec2:CreateTags on the team key for anyone outside\nyour platform-admin role. Otherwise \"scoped by tag\" degrades into\n\"scoped by tag until someone removes the tag,\" which isn't a security\nboundary at all — it's a labeling convention people are trusting each\nother to respect.
aws:SourceIp restricts API calls to a CIDR range — your office IP, your\nVPN egress, or (less usefully) 0.0.0.0/0. It's most valuable on IAM users\nor roles that still authenticate outside of AWS-managed network paths\n(a CI runner with a static egress IP, a break-glass admin user):
{\n \"Effect\": \"Deny\",\n \"NotAction\": [\"iam:ChangePassword\", \"iam:GetUser\"],\n \"Resource\": \"*\",\n \"Condition\": {\n \"NotIpAddress\": {\n \"aws:SourceIp\": [\"203.0.113.0/24\", \"198.51.100.0/24\"]\n },\n \"Bool\": {\n \"aws:ViaAWSService\": \"false\"\n }\n }\n}\nTwo things worth calling out. First, aws:SourceIp looks at the caller's\npublic IP for calls made directly to AWS, but for calls made from inside a\nVPC through a VPC endpoint, you want aws:VpcSourceIp instead — it's the\nprivate IP inside the VPC, and it only populates when the call actually\ntransits a VPC endpoint. Mixing the two up is a common reason a\nnetwork-scoped policy silently fails for endpoint traffic: the condition\nkey you checked simply isn't present on that request, so it evaluates as\n\"condition not met\" and the whole statement is skipped.
Second, that aws:ViaAWSService: false condition isn't decoration —\nwithout it, this Deny also blocks AWS services calling APIs on your\nbehalf (e.g., CloudFormation invoking IAM during a stack operation),\nbecause those calls don't carry your IP at all and would otherwise fail\nthe NotIpAddress check and get denied. This is the single most common\nway I've seen an IP-restriction policy break someone's CI pipeline the\nday after it ships.
For your most sensitive actions — deleting a CloudTrail trail,\ndeactivating GuardDuty, changing another user's credentials — require not\njust \"authenticated\" but \"authenticated with MFA, recently\":
\n{\n \"Effect\": \"Deny\",\n \"Action\": [\n \"iam:DeleteUser\",\n \"iam:DeleteRole\",\n \"iam:UpdateAccessKey\",\n \"cloudtrail:StopLogging\",\n \"guardduty:DeleteDetector\"\n ],\n \"Resource\": \"*\",\n \"Condition\": {\n \"BoolIfExists\": {\n \"aws:MultiFactorAuthPresent\": \"false\"\n }\n }\n}\nUse BoolIfExists rather than a plain Bool here. aws:MultiFactorAuthPresent\nis simply absent from the request context for some call types (notably\ncalls made with temporary credentials from certain federation flows, or\nservice-to-service calls), and a plain Bool condition against a missing\nkey evaluates as false — which, counterintuitively, means the Deny\ncondition (\"aws:MultiFactorAuthPresent\": \"false\") matches and the action\ngets denied even for legitimate non-interactive callers. BoolIfExists\nonly evaluates the condition when the key is present, so it doesn't\nmisfire on requests where MFA presence genuinely isn't reportable.
Add aws:MultiFactorAuthAge (in seconds) if you want to force\nre-authentication for stale sessions rather than trusting an MFA check\nfrom ten hours ago:
\"Condition\": {\n \"NumericGreaterThan\": {\n \"aws:MultiFactorAuthAge\": \"3600\"\n }\n}\nThe pattern above — Effect: Deny plus NotAction — is powerful but easy\nto get backwards. NotAction in a Deny statement means \"deny everything\nexcept these actions,\" so the actions you list are the ones exempted\nfrom the deny, not the ones targeted by it. I've seen policies where\nsomeone wanted to restrict a set of dangerous actions and reached for\nNotAction, accidentally exempting exactly the actions they meant to\nlock down while denying everything else in the account. If your intent is\n\"deny these specific actions unless X,\" use a plain Action list. Reserve\nNotAction for \"deny everything except these few things\" — the safe\nsubset you're carving out, like the two IAM self-service calls in the\naws:SourceIp example above, which a locked-out user still needs to fix\ntheir own password.
The other non-obvious trap: explicit Deny always wins, but a missing\ncondition key doesn't always mean \"deny.\" Whether an absent key trips\nthe condition depends entirely on which condition operator you used\n(Bool vs BoolIfExists, StringEquals vs StringEqualsIfExists). IAM\ndoesn't warn you when a policy's condition silently never matches because\nthe key isn't in the request context — it just evaluates false and moves\non, and the action proceeds under whatever your next-most-permissive\nstatement allows. Test conditional Deny policies with the IAM Policy\nSimulator against the actual principal and a realistic set of request\nparameters before you rely on them, not just against a policy you're\nreading and assuming is correct.
Don't attach a new condition-scoped Deny directly to a broad group and\nwalk away. Roll it out the way I roll out any access-tightening change:\nattach it to a single test principal first, run the actual workflows that\nprincipal needs (including CI and automation, not just interactive\nconsole use), check CloudTrail for errorCode: AccessDenied events you\ndidn't expect, and only then widen the attachment to the group or\naccount-wide SCP. Tag-based and MFA-based conditions are cheap to write\nand expensive to debug in production if you get the IfExists variant\nwrong — the fifteen minutes in the policy simulator is worth it.
For as long as I've been writing Terraform, \"S3 backend\" meant \"S3 backend\nplus a DynamoDB table for locking,\" full stop. You'd provision a tiny\npay-per-request table, grant your CI role dynamodb:GetItem/PutItem/\nDeleteItem on it, and never think about it again except when someone asked\n\"wait, why do we have a DynamoDB table for a static site's infra?\" As of\nTerraform 1.10, that table is no longer required — S3 itself can do the\nlocking, using conditional writes. This site's own terraform/backend.hcl\nruns on exactly this setup, no DynamoDB table anywhere in the account. Here's\nthe history, the config, and the migration path if you're still running the\nold way.
State locking exists to stop two terraform apply runs from racing each\nother and corrupting state — classic \"last writer wins\" data loss. S3 never\nhad a native compare-and-swap primitive, so Terraform's S3 backend\npiggybacked on DynamoDB's conditional PutItem (attribute_not_exists) to\nimplement a lock: acquire a lock by writing an item keyed on the state path,\nrelease it by deleting the item. It worked well, but it meant every S3\nbackend needed a second AWS service, a second IAM policy, and a second thing\nto provision correctly before your very first terraform init — plus a\nmanual dynamodb:DeleteItem to clean up a stuck lock after a killed CI job,\nwhich everyone running Terraform in CI has done at least once.
Amazon added conditional writes to S3 itself in August 2024\n(If-None-Match / If-Match support on PutObject), and HashiCorp shipped\nsupport for using that directly as a locking mechanism in Terraform\n1.10 (November 2024) via a new backend argument: use_lockfile. No\nDynamoDB, no second service, no separate IAM policy — the same S3 permissions\nyour state already needed now cover locking too.
This is the actual backend block from this repo, split the way Terraform\nbackends normally are: the non-secret pieces in a partial config file\n(terraform/backend.hcl), and the empty backend \"s3\" {} stanza in\nversions.tf so nothing hardcoded ends up needing per-environment variables.
# terraform/versions.tf\nterraform {\n required_version = \">= 1.10\"\n\n required_providers {\n aws = {\n source = \"hashicorp/aws\"\n version = \">= 5.60, < 8.0\"\n }\n }\n\n # Remote state in S3 with native locking (Terraform >= 1.10, no DynamoDB).\n # Non-secret values are supplied at init time via backend.hcl.\n backend \"s3\" {}\n}\n# terraform/backend.hcl\nbucket = \"jstgtech-web-tfstate\"\nkey = \"site/terraform.tfstate\"\nregion = \"us-east-1\"\nencrypt = true\nuse_lockfile = true\nterraform init -backend-config=backend.hcl\nThat's the whole thing. use_lockfile = true tells the S3 backend to write a\n<key>.tflock object next to your state object during plan/apply, using\na conditional PutObject so a second concurrent run gets a hard failure\ninstead of a silent overwrite, and deletes the lock object when the run\nfinishes (or on terraform force-unlock if a run got killed before it could\nclean up). No dynamodb_table argument anywhere — the S3 backend has\nsupported that argument for years for the old locking path, and you can\ntechnically still set both during a transition (more on that below), but a\nfresh setup like this one just doesn't need it.
apply against a bad plan or you need to roll back a\ncorrupted state file. use_lockfile doesn't touch this at all — you still\nneed aws_s3_bucket_versioning with status = \"Enabled\" on the bucket, the\nsame as the DynamoDB-locking days. This repo's bootstrap layer\n(terraform/bootstrap/main.tf) sets it explicitly: resource \"aws_s3_bucket_versioning\" \"tfstate\" {\n bucket = aws_s3_bucket.tfstate.id\n versioning_configuration {\n status = \"Enabled\"\n }\n }\ns3:PutObject and s3:DeleteObject on the state bucket — most roles that\ncould already read/write state already have this, but if you'd scoped a\nrole down to s3:GetObject + s3:PutObject on just the state key, tighten\nit to cover the <key>.tflock path too (or just the whole prefix, which is\nsimpler and what most setups do — s3:* scoped to the bucket ARN, not\nwildcarded across accounts).backend block cannot reference variables, locals, or anything\n** computed — this predates native locking but people relearn it every time.**\nYou cannot do backend \"s3\" { bucket = var.state_bucket }. That's why\nbackend \"s3\" {} is empty in versions.tf and everything lives in\nbackend.hcl, passed at init time with -backend-config. If you have\nmultiple environments, you'll have multiple .hcl files (or a templated\none generated by CI before init), not variables inside the block.use_lockfile needs Terraform ≥ 1.10, checked hard. If someone on your\nteam is still on 1.9 (or an old pinned CI container), init will just\nerror on the unrecognized argument. Bump required_version in the same PR\nthat adds use_lockfile so the mismatch fails loud at init instead of\nsomeone silently running an older binary against a backend config it\ndoesn't understand.aws dynamodb delete-item the lock row by hand, or use\nterraform force-unlock <LOCK_ID> which did the same thing under the\nhood. With native locking, force-unlock still works — Terraform reads the\nlock ID from the .tflock object's contents — but if you're ever debugging\nby hand, you're looking for an S3 object, not a DynamoDB item, and it's\nnamed <key>.tflock next to your actual state object in the same bucket.If you've got a working dynamodb_table setup today, don't just delete the\ntable and flip use_lockfile on in the same change — do it in two steps so\nyou're never in a state where in-flight infra changes could hit a locking\nconfig mismatch:
use_lockfile = true to backend.hcl alongside the existing\ndynamodb_table entry**, then run: terraform init -reconfigure\n-reconfigure (not -migrate-state) is correct here — you're not moving\nstate to a new bucket/key, you're just changing backend configuration on\nthe same backend type. Terraform will pick up native locking going\nforward. At this point both mechanisms are technically configured, but\nTerraform only uses one lock mechanism per backend version — as of the\n1.10–1.13 line, setting both dynamodb_table and use_lockfile together\nis explicitly supported as a transition state precisely for this\nmigration.
plan and apply to confirm the new lock path works —\nwatch for the .tflock object appearing in the state bucket during the\nrun (aws s3 ls s3://your-tfstate-bucket/ --recursive | grep tflock\nwhile it's running, or just check after a plan that holds the lock long\nenough).dynamodb_table from backend.hcl and\nrun terraform init -reconfigure again. Nothing about your state file\nchanges in this step — you're purely dropping a backend config key.terraform state rm/destroy it via whatever provisioned it\noriginally (if the table itself was Terraform-managed in a bootstrap\nlayer, remove the resource block and apply; if it was created by hand,\njust delete it from the console or CLI). For a low-traffic project this is\nmaybe $1–2/month in the pay-per-request billing mode most people use for\nlock tables, so the savings are more about **one fewer resource to\nprovision, IAM-scope, and explain to the next person reading your\nTerraform** than meaningful dollars — the real win is operational, not\nfinancial.I wouldn't rip it out reflexively everywhere:
\nuse_lockfile simply isn't available..tflock objects the way tools built around the DynamoDB API might.For a single-account, single-tool setup like this site's — GitHub Actions\nrunning terraform plan/apply via OIDC, nothing else touching state — none\nof those apply, which is exactly why terraform/backend.hcl here has no\ndynamodb_table line at all.
Every team building on top of a foundation model eventually asks the same\nquestion: call the provider's API directly, or go through a managed layer.\nBedrock's answer is \"stay inside your AWS account boundary and get one API\nacross multiple model providers\" — worth understanding precisely, since\n\"it's just an API wrapper\" undersells what it actually replaces, and\noverselling it as \"AWS's AI service\" undersells how much is still on you.
\nBedrock is a managed access layer to foundation models from multiple\nproviders — Anthropic's Claude family, Meta's Llama, Amazon's own Nova\nmodels, Mistral, and others — through one consistent API and SDK, with no\nGPU infrastructure for you to provision or manage. You're not running a\nmodel; you're calling one that AWS hosts, billed per request. The value\nproposition over calling a provider directly is staying inside your existing\nAWS security boundary: IAM for auth instead of a separate API key to\nrotate and store, VPC endpoints so traffic never leaves AWS's network,\nCloudTrail logging every invocation, and data that (per AWS's terms) isn't\nused to train underlying models — which matters if your compliance posture\nalready assumes AWS's shared responsibility model and adding a new\nthird-party vendor relationship is its own review cycle.
\nGuardrails for Amazon Bedrock is a configurable content-filtering layer\nyou attach to any model call, independent of which underlying model you're\nusing. It can block or mask categories of content (hate speech, violence,\nprompt-injection attempts), redact detected PII in both the prompt and\nthe response (SSNs, credit card numbers, emails — configurable per field),\nenforce topic denial (block responses about specific configured topics\nentirely — \"don't discuss competitor products,\" for instance), and run a\ncontextual grounding check that scores a RAG response against its source\ndocuments to catch hallucination before it reaches the user. It runs as a\npolicy applied at invocation time, which means you can update filtering\nrules without redeploying application code or retraining anything.
\nThe honest caveat: guardrails are pattern- and classifier-based, not a\nguarantee. They meaningfully reduce the surface area for bad outputs and\ngive you an audit trail of what was blocked, but they're a mitigation layer,\nnot a substitute for output validation in anything security- or\ncompliance-sensitive downstream.
\nBedrock bills per token, in and out, and pricing varies by model — a\nlarger, more capable model can be an order of magnitude more expensive per\ntoken than a smaller one for the same request. The trap is defaulting every\ncall in an application to the largest/newest model \"because it's the best,\"\nwhen a cheaper model handles the bulk of routine requests (simple\nclassification, short summarization) just as well. Provisioned\nThroughput (reserved capacity, billed hourly regardless of usage) is worth\nit only at sustained high volume; below that threshold, on-demand per-token\npricing is cheaper even though the per-unit rate looks higher, because\nyou're not paying for idle reserved capacity between requests.
\nRoute by task complexity instead of hardcoding one model for an entire\napplication — a cheap, fast model for classification/extraction/short\nresponses, a larger model reserved for requests that actually need deeper\nreasoning. Bedrock's consistent API across model families makes that routing\na config decision rather than a rewrite, which is most of the point of using\na multi-model layer instead of committing to one provider's SDK directly.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "aws", "bedrock", "ai", "spotlight" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-amazon-eventbridge/", "url": "https://jstgtech.com/blog/2026-08-10-amazon-eventbridge/", "title": "Service spotlight: Amazon EventBridge beyond Lambda glue", "summary": "What EventBridge actually buys you over SNS/SQS point-to-point wiring, how schema discovery and archive/replay work, and where rule limits bite.", "content_html": "Most teams meet EventBridge as \"the thing that triggers a Lambda on a cron\nschedule\" and stop there. That's a legitimate use, but it undersells the\nservice — EventBridge is an event bus, and the pattern-matching/routing layer\nunderneath it is worth understanding on its own, separate from the scheduler\nfeature that happens to share the console.
\nAn event bus is a named channel that receives JSON events and evaluates\nthem against a set of rules. Each rule has an event pattern — a partial\nJSON match against the event's fields — and a list of targets (Lambda, SQS,\nStep Functions, Kinesis, another event bus, over 20 AWS services directly).\nWhen an event matches, EventBridge fans it out to every matching rule's\ntargets, in parallel, with built-in retry and an optional dead-letter queue\nper target. You publish an event once; the bus decides who cares.
\nEvery account gets a default bus that also receives events natively from\n~200 AWS services (an S3 upload, an ECS task state change, a CodePipeline\nstage transition) with zero integration code — that's the part people miss.\nYou can also create custom buses for your own application events, and\npartner buses for SaaS integrations (Datadog, PagerDuty, Zendesk) that\npush events directly into your account.
\nThe alternative to EventBridge is usually \"Lambda A calls Lambda B directly\"\nor \"publish to an SNS topic with N subscriptions.\" Both work, but they\ncouple the publisher to knowing who's listening. EventBridge inverts that:\nthe publisher emits one event describing what happened, and consumers\ndeclare what they care about via pattern matching, without the publisher's\ncode ever changing when a fourth consumer shows up. Adding \"also send a\nSlack notification when an order ships\" is a new rule, not a code change to\nthe order service.
\nSchema discovery is the other underused feature: point EventBridge at a\nbus and it infers a schema registry from the events flowing through it,\nincluding versioning as the shape evolves, and can generate strongly-typed\nbindings (Java, Python, TypeScript) for consumers. For anything beyond a\nhandful of hand-maintained event types, that beats a wiki page describing\n\"here's what fields this event has, trust me.\"
\nEvery bus can have an archive attached — EventBridge stores a copy of\nmatching events (with a retention period you set, including indefinite) and\ncan replay them into the bus later, re-triggering the same rules. That's\nthe feature that turns \"we had a bug in the order-confirmation-email Lambda\nfor six hours\" from a data-backfill script into a console button: fix the\nLambda, replay the archived window, done. It's also genuinely useful for\ntesting — replay production events into a rule pointed at a dev target\ninstead of hand-crafting test payloads.
\nEach event pattern rule can have at most 5 targets, and a bus can have\nup to 300 rules by default (a soft limit, raisable). Teams that treat\nEventBridge as a single giant bus with dozens of broad-matching rules and\nlong target lists eventually hit both limits and end up debugging which of\n15 similar-looking rules actually matched a given event — the console's\n\"test pattern\" tool helps, but the real fix is scoping rules narrowly by\ndetail-type and source from the start rather than one catch-all rule per\nconsumer with a growing target list.
Custom events published to a custom bus are billed per million events;\nevents matched from AWS service integrations on the default bus are free\n— only your own published events cost anything. Archive storage is billed\nseparately by GB, and replays re-invoke targets, so replaying a large\narchived window against a Lambda target bills exactly like the original\ntraffic did. It's cheap at normal volumes, but a botched pattern that\naccidentally matches everything (an empty or overly broad source filter)\nand fans out to an expensive target is the way people get a surprise line\nitem, same as any fan-out system.
Give every custom event a detail-type and source from day one, even for\na single-consumer event — retrofitting rule scoping after five teams are\npublishing to the same bus with inconsistent field names is far more painful\nthan establishing the convention up front. If you're publishing from\nmultiple services, agree on a naming scheme (source: \"orders.checkout\",\ndetail-type: \"OrderPlaced\") before the first event ships, not after.
SQS is old, boring, and one of the most reliable services AWS runs — which\nis exactly why it's easy to wire up carelessly and not notice until a queue\nbacks up in production. The service itself rarely fails; the mistakes are\nalmost always in how visibility timeouts, DLQs, and consumer scaling are\nconfigured around it.
\nA producer sends a message; a consumer polls, receives it, processes it, and\nexplicitly deletes it. Between receive and delete, the message is\ninvisible to other consumers for the visibility timeout duration —\nnot deleted, just hidden, so a second consumer doesn't pick up the same\nmessage while the first is still working it. If the consumer crashes,\ntimes out, or never calls delete, the message reappears after the timeout\nexpires and gets redelivered. That reappear-on-failure behavior is the\nentire reliability model, and most SQS production issues trace back to a\nmismatch somewhere in that loop.
\nIf your consumer takes 90 seconds to process a message but the queue's\nvisibility timeout is set to the default 30 seconds, the message becomes\nvisible again while it's still being processed — a second consumer\npicks it up, and now you're processing the same message twice concurrently.\nFor anything non-idempotent (charging a card, sending an email, decrementing\ninventory), that's a correctness bug, not just wasted compute. Set the\nvisibility timeout to comfortably exceed your maximum expected\nprocessing time, not the average — or use ChangeMessageVisibility to\nextend it dynamically from inside a long-running handler if processing time\nvaries widely.
A dead-letter queue catches messages that fail processing repeatedly (by\nmaxReceiveCount) instead of retrying forever and blocking the queue behind\none poison message. Skipping a DLQ means a single malformed message can\nloop indefinitely, burning consumer capacity on retries that will never\nsucceed. But the more common failure is having a DLQ and never alarming on\nit — messages quietly pile up, nobody notices for weeks, and by the time\nsomeone checks, the redrive window (or business relevance of the messages)\nhas passed. A DLQ without a CloudWatch alarm on ApproximateNumberOfMessages\nVisible is a DLQ that isn't doing its job.
ApproximateNumberOfMessagesVisible tells you how many messages are\nwaiting, but on its own it doesn't tell you whether that's a processing\noutage or just a traffic spike your consumers will burn down in ten\nminutes. Pair it with ApproximateAgeOfOldestMessage — a queue that's deep\nbut where the oldest message is only 90 seconds old is healthy and\nscaling; a queue where the oldest message is 40 minutes old means something\ndownstream is actually stuck, regardless of current depth. Age-of-oldest is\nusually the better alarm trigger for \"something is broken\" versus\n\"we're busy.\"
Standard queues are at-least-once delivery with best-effort ordering and\neffectively unlimited throughput — the right default for most workloads,\nand your consumer logic needs to be idempotent regardless (duplicate\ndelivery is a normal, expected occurrence, not an edge case). FIFO\nqueues add exactly-once processing and strict ordering **within a message\ngroup**, at the cost of a throughput ceiling (3,000 messages/sec with\nbatching, per API action) and higher per-request cost. Reach for FIFO only\nwhen ordering is a genuine correctness requirement (e.g., applying account\nbalance changes in sequence) — defaulting to FIFO \"to be safe\" trades away\nthroughput headroom for a guarantee most workloads don't actually need.
\nSet a redrive policy that moves messages back from the DLQ to the source\nqueue (SQS supports this natively now, no custom script needed) as part of\nyour incident-response runbook, not as something you improvise during an\noutage. Deciding in advance how many retries a message deserves before\nlanding in the DLQ, and what \"redrive after the bug is fixed\" actually looks\nlike operationally, is a five-minute conversation before go-live and a much\nworse one at 2am with a few thousand stuck messages.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "aws", "sqs", "spotlight" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-aurora-serverless-v2/", "url": "https://jstgtech.com/blog/2026-08-10-aurora-serverless-v2/", "title": "Service spotlight: Aurora Serverless v2 without the v1 cold starts", "summary": "How Aurora Serverless v2 scales capacity in fine-grained ACUs without pausing, where it still costs more than provisioned Aurora, and its real limits.", "content_html": "Aurora Serverless v1 had a reputation problem, and it was earned: scale-to-\nzero meant a cold start that could take tens of seconds on the next\nconnection, and scaling itself worked in coarse steps that didn't handle\nsudden bursts well. Enough people got burned that \"Aurora Serverless\" became\nshorthand for \"don't.\" v2 is a different architecture, not a patched v1, and\nit's worth revisiting that reputation now that the two are easy to conflate.
\nv1 scaled by swapping the entire database to a differently-sized instance\nbehind the scenes — a real cutover with a brief connection drop, done in\ndiscrete capacity steps. v2 scales capacity in place, in increments of\n0.5 ACUs (Aurora Capacity Units, each roughly 2 GiB of memory plus\nproportional CPU and networking), typically in under a second, with no\nconnection interruption. You set a min and max ACU range per cluster (as low\nas 0, up to 256 per instance) and Aurora adjusts within it continuously\nbased on actual load — CPU, memory, and active connections — rather than\nstepping through a small number of predefined sizes.
\nCrucially, v2 instances can participate in a Global Database, use\nMulti-AZ with the same failover mechanics as provisioned Aurora, and\neven mix with provisioned instances in the same cluster (serverless reader,\nprovisioned writer, or vice versa) — none of which v1 supported. That mixed-\ninstance-class capability is the feature that makes it viable for real\nproduction topologies instead of just dev/test databases.
\nr6g.xlarge and resizing later (a\ndisruptive operation on provisioned Aurora), set a wide ACU range and let\nthe actual traffic tell you where it settles — then, if it settles at a\nsteady high number, consider whether provisioned pricing would now be\ncheaper for that stable load.For a steady, predictable, high-utilization workload, provisioned Aurora\nwith Reserved Instance pricing is cheaper than v2 at the equivalent\ncapacity — the per-ACU-hour rate carries a premium over the equivalent\nprovisioned instance-hour, same trade as Fargate versus EC2. If your\ndatabase sits at a consistent 8 ACUs of load 24/7 with no meaningful\nvariance, you're paying for elasticity you're not using.
\nThere's also a real floor: 0.5 ACU minimum for an \"always-on\" cluster is\nroughly comparable to a small t4g instance's baseline cost — v2 doesn't\nscale to true zero like v1 could (v2 got a scale-to-zero option later, but\nit reintroduces a cold-start pause on the next connection, the exact\ntrade-off v2 was built to avoid at higher tiers, so it's really only sane\nfor genuinely idle dev environments).
Scaling is fast but not instant, and it reacts to load — it doesn't\npredict it. A workload that goes from idle to a hard spike in under a\nsecond (a flash-sale-style traffic burst) can briefly get throttled or see\nelevated latency while Aurora scales up to meet it, because there's a real\ncontrol loop with a reaction time, not a pre-provisioned buffer sitting\nready. Setting a higher minimum ACU than \"what average load needs\" gives\nthe scaling logic headroom to absorb bursts before it has to react, at the\ncost of paying for that headroom continuously — a direct latency-versus-cost\nknob, not a free win.
\nWatch the ServerlessDatabaseCapacity CloudWatch metric against your min/\nmax bounds for a couple of weeks under real traffic before treating the\nrange as tuned. A cluster that's pinned at its max ACU most of the day isn't\n\"elastic,\" it's a provisioned database paying the serverless premium —\nthat's the signal to either raise the max or move to provisioned capacity\nfor the baseline and reserve v2 for genuinely variable secondary workloads.
GraphQL servers usually mean running (and scaling, and patching) a Node\nprocess that resolves fields by calling out to your actual data sources.\nAppSync's pitch is skipping that process entirely — you define a schema and\nper-field resolvers that AppSync itself executes, with no server of yours in\nthe request path unless a resolver specifically needs one. That's a\nmeaningfully different operational model, and it comes with a cost model\nthat's easy to underestimate if you're used to thinking in request-per-\nsecond server pricing.
\nYou upload a GraphQL schema and attach a resolver to each field that\nneeds one — a small piece of logic that maps the incoming GraphQL selection\nto a call against a data source: DynamoDB, Aurora (via RDS Data API),\nOpenSearch, EventBridge, HTTP endpoints, or a Lambda function for anything\ncustom. Resolvers used to be written in VTL (Apache Velocity Template\nLanguage, borrowed from API Gateway's mapping templates) — verbose,\nunfamiliar syntax most teams didn't already know. AppSync now supports\nJavaScript resolvers as the default recommended option, which is a\nmeaningfully better developer experience if you're starting fresh; VTL still\nworks for existing resolvers and some advanced batching patterns.
\nThe other core piece is built-in real-time subscriptions over\nWebSockets — a client subscribes to a mutation, and AppSync pushes the\nresult to every subscribed client when it fires, with the pub/sub\ninfrastructure entirely managed. That's the feature that's genuinely hard to\nreplicate cheaply with a hand-rolled server; it's Socket.io-style\ninfrastructure you don't have to run.
\nIf most of your resolvers end up calling a Lambda anyway (because the logic\nis too custom for a direct data-source resolver), you've mostly rebuilt\n\"API Gateway plus Lambda\" with GraphQL schema validation on top — worth it\nif GraphQL's client-side benefits (one round trip, client-specified shape)\nmatter to you, not worth it if you just wanted a managed API layer. And\nteams unfamiliar with GraphQL's N+1 problem will hit it here exactly like\nanywhere else: a nested field resolver that fires once per parent item\nneeds batching (AppSync's batch invoke for Lambda resolvers, or DynamoDB\nBatchGetItem) or it'll quietly turn one query into hundreds of downstream\ncalls.
\nAppSync bills per query/mutation request and, separately, per\nresolver invocation within that request — a query resolving five nested\nfields is one request but potentially five (or more, with N+1) billed\nresolver executions, plus real-time subscription connection-minutes and\nmessage counts on top if you're using them. A single GraphQL query that\nlooks simple from the client can fan out into a surprising number of billed\nresolver calls server-side; the AWS docs' pricing page undersells how fast\nthat adds up on a deeply nested schema, and it's worth actually tracing a\nrepresentative query's resolver count before assuming the bill will track\nrequest count linearly.
\nUse pipeline resolvers to chain multiple resolution steps (an auth\ncheck, then a DynamoDB read, then a transform) into a single resolver\nattached to one field, rather than pushing that logic into nested field\nresolvers on the schema — it collapses what would be several billed\nresolver invocations per query into fewer, more predictable ones, and keeps\nthe N+1 pattern from creeping in as the schema grows.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "aws", "appsync", "graphql", "spotlight" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-aws-backup/", "url": "https://jstgtech.com/blog/2026-08-10-aws-backup/", "title": "Service spotlight: AWS Backup replaces per-service snapshot scripts", "summary": "How AWS Backup centralizes EBS, RDS, DynamoDB, and EFS backup policy into one plan, and what it still leaves you to configure yourself.", "content_html": "Every AWS account I've inherited that predates AWS Backup has the same\nshape: a Lambda that snapshots EBS volumes on a cron, a separate RDS\nautomated backup window nobody's checked in years, and DynamoDB\npoint-in-time recovery toggled on for some tables and not others, with no\nsingle place to see whether any of it is actually working. AWS Backup\nexists to replace that patchwork with one policy.
\nAWS Backup is a managed service that centralizes backup scheduling,\nretention, and recovery across most stateful AWS resource types — EBS,\nRDS/Aurora, DynamoDB, EFS, FSx, Storage Gateway volumes, and EC2 instances\nas a whole (not just their volumes) — under one backup plan. A plan\ndefines a schedule (cron expression), a retention period, and a\nlifecycle that can transition backups to cold storage and eventually\nexpire them, then you assign resources to the plan by tag or by\nexplicit resource ID. Everything the plan touches shows up in one backup\nvault with a consistent job history, so \"did last night's backup\nactually succeed\" is one console view instead of checking five different\nservices' own backup mechanisms.
\nbackup: daily, assign that tag to a plan, and every new\nEBS volume or RDS instance with that tag is covered automatically going\nforward — no per-resource setup step anyone can forget.AWS Backup schedules and retains backups; it doesn't validate that a\nrestore actually works. Restore testing is a separate feature (Restore\nTesting plans) that you have to opt into explicitly and point at real\nresources — untested backups are a common source of \"we had backups but\nthe restore didn't work\" incidents, and AWS Backup existing doesn't change\nthat unless you actually run the restore tests.
\nIt also doesn't replace application-consistent backup logic for\ndatabases that need it. RDS/Aurora snapshots through AWS Backup use the\nsame underlying mechanism as native RDS snapshots (so they're\ncrash-consistent, and for most engines that's sufficient), but if you're\nbacking up something like a self-managed database on EC2 that needs\nquiesce-before-snapshot logic, AWS Backup's EC2/EBS support won't give you\nthat for free — you still need your own pre-snapshot hooks.
\nCosts are backup storage (billed per GB-month, warm vs cold tier priced\ndifferently) plus, for some resource types, restore costs. The lifecycle\ntransition to cold storage (available for EBS, RDS, DynamoDB, and a few\nothers) meaningfully cuts long-retention costs, but cold-tier backups\ntypically have a minimum retention period before transition and can\ncarry early-deletion or restore-time costs, so a plan built for\n\"retain 7 years, transition to cold at 90 days\" needs those constraints\nmodeled, not just the headline GB-month price.
\nTurn on AWS Backup Audit Manager (a separate but related feature) once\nyou have plans running — it continuously evaluates whether resources are\nactually covered by a backup plan matching your organization's policy and\nflags drift, which catches the \"someone launched a new RDS instance\nwithout the required tag\" gap that a plan alone won't.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "aws", "backup", "disasterrecovery", "spotlight" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-aws-config/", "url": "https://jstgtech.com/blog/2026-08-10-aws-config/", "title": "Service spotlight: AWS Config for compliance-as-code and drift detection", "summary": "How AWS Config records resource configuration history and evaluates it against rules continuously, and why it pairs with — not replaces — Terraform.", "content_html": "Terraform tells you what infrastructure should look like at apply time.\nIt doesn't tell you what changed at 3am when someone with console access\nmanually flipped a security group rule to unblock themselves and forgot to\nrevert it. That gap — configuration drift between what you declared and\nwhat's actually running — is what AWS Config is built to close.
\nAWS Config continuously records the configuration state of supported\nresources in your account (over 300 resource types) as a timestamped\nhistory, and evaluates that state against Config rules — either AWS\nmanaged rules (s3-bucket-public-read-prohibited,\nrestricted-ssh, iam-password-policy) or custom rules backed by your\nown Lambda function or Guard/CloudFormation Guard policy. Every recorded\nchange produces a configuration item, and Config keeps a full\ntimeline, so you can ask \"what did this security group's rules look like\nat 2:14pm last Tuesday\" and get an actual answer, not a guess reconstructed\nfrom CloudTrail events.
Rules run either on a schedule (periodic) or triggered by a configuration\nchange (change-triggered), and each evaluation produces a\ncompliant/non-compliant verdict per resource that shows up in one\ndashboard across the account — or, aggregated, across an entire\nOrganization via Config aggregators.
\nConfig is observational and reactive — it tells you what's true now\nand evaluates it against policy, but it doesn't prevent a bad change from\nhappening in the first place the way a Terraform plan/apply gate or an SCP\ndoes. And Terraform's state file, while it also describes resource\nconfiguration, only reflects what Terraform itself last applied — it goes\nstale the moment something changes outside Terraform, which is exactly\nthe blind spot Config is designed to catch. The two are complementary:\nTerraform (plus policy-as-code gates like Checkov) prevents drift at\napply time; Config catches whatever gets through anyway, including\nchanges that never went through Terraform at all.
\nConfig bills per configuration item recorded and per **rule\nevaluation**, both of which scale with account activity and resource\ncount — a large, churny account with many resource types and frequent\nchanges will rack up config-item costs faster than a small, stable one.\nTurning on recording for every supported resource type across every\naccount in an Organization without checking projected volume first is the\nmost common way a Config bill surprises someone; scope the recorder to\nresource types you actually intend to have rules for if cost is a\nconcern, rather than defaulting to \"record everything.\"
\nStart with a conformance pack for a benchmark you already care about\n(the AWS-provided CIS or Foundational Security Best Practices packs are\ngood defaults) instead of hand-picking individual managed rules — it gets\nyou broad, curated coverage immediately, and you can prune or add rules\nfrom there once you see what's actually flagging in your account.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "aws", "config", "compliance", "spotlight" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-aws-fargate/", "url": "https://jstgtech.com/blog/2026-08-10-aws-fargate/", "title": "Service spotlight: AWS Fargate and its per-task pricing surprises", "summary": "What you actually give up going serverless with Fargate over EC2-backed ECS, how per-task vCPU/memory billing adds up, and where Fargate Spot fits.", "content_html": "\"Just use Fargate\" is the default answer to \"how do I run this container\"\noften enough that it's worth being precise about what it trades away, not\njust what it saves. It's not free serverless magic — it's EC2 capacity\nmanagement moved from your team to AWS, billed per task instead of per\ninstance, and that shift changes both your ops burden and your cost curve.
\nFargate is a launch type for ECS (and EKS) that removes the EC2 instance\nlayer entirely. With the EC2 launch type, you manage a cluster of instances,\nsize them, patch them, and bin-pack tasks onto them yourself (or via Cluster\nAutoscaler-style capacity providers). With Fargate, you specify vCPU and\nmemory per task definition, and AWS runs each task on its own\nright-sized, isolated compute — no instances to see, patch, or pack. You\nstill define everything else about the container (image, environment,\nnetworking, IAM task role) exactly like EC2-backed ECS; only the compute\nlayer underneath changes.
\nFargate's per-vCPU-hour and per-GB-hour pricing is meaningfully higher than\nthe equivalent on-demand EC2 price for the same resources — AWS is charging\nfor not having to manage the instance, and that premium is real. For\nsteady-state, high-utilization workloads (a fleet of API servers running\n24/7 at 60-80% CPU), a well-managed EC2-backed ECS cluster with Reserved\nInstances or Savings Plans underneath is usually meaningfully cheaper for\nthe same compute — you're just paying someone (your own team) in ops time\ninstead of paying AWS in margin. The crossover point depends on utilization\nand how much that ops time actually costs, but \"always Fargate\" and \"always\nEC2\" are both wrong defaults; it's a per-workload call.
\nFargate also has hard ceilings EC2 doesn't: max 4 vCPU / 30 GB memory per\ntask on the standard configuration tier (higher limits exist but require\nopt-in and aren't universally available), no GPU support, and no control\nover the underlying kernel or host-level tuning (sysctls, huge pages) that\nsome latency-sensitive workloads need.
\nFargate Spot runs tasks on spare capacity at up to a 70% discount versus\non-demand Fargate pricing, with the same two-minute interruption warning\nmodel as EC2 Spot. It's a good fit layered into an ECS capacity\nprovider strategy — e.g., a base count of on-demand Fargate tasks to\nguarantee minimum capacity, with everything above that scaled on Fargate\nSpot. That works well for stateless, horizontally-scaled services behind a\nload balancer where losing one task briefly just means the ALB stops\nrouting to it and ECS replaces it; it's a poor fit for long-running batch\njobs that don't checkpoint, since an interruption mid-job means starting\nover.
Fargate bills per-task at 1-second granularity with a 1-minute minimum,\nand — this is the part people miss — vCPU and memory are billed\nindependently at their configured amounts, not at actual usage. A task\ndefined with 2 vCPU / 4 GB but that only ever uses 0.5 vCPU still bills for\nthe full 2 vCPU the whole time it's running. Over-provisioning task\ndefinitions \"to be safe\" is the single most common way Fargate bills come in\nhigher than expected — right-size against actual CloudWatch Container\nInsights utilization, not a guess, and revisit it after the workload has run\nfor a few weeks under real traffic.
\nIf you're running dozens of small, short-lived tasks, check whether\nECS Service Connect or batching multiple containers into one task\ndefinition (as sidecars sharing the task's vCPU/memory allocation) reduces\ntotal billed task-time versus one task per container — the per-task minimum\nbilling granularity means many tiny tasks can cost more in aggregate than\nfewer, right-sized ones doing the same work.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "aws", "fargate", "ecs", "spotlight" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-aws-step-functions/", "url": "https://jstgtech.com/blog/2026-08-10-aws-step-functions/", "title": "Service spotlight: AWS Step Functions for real workflows", "summary": "When AWS Step Functions earns its keep over a pile of Lambda glue code, where its per-transition pricing bites, and a Map state trick worth knowing.", "content_html": "Every few months I see a team reinvent Step Functions badly: a Lambda that\ninvokes another Lambda, wrapped in a try/except that retries three times\nand then writes a row to DynamoDB so a cron job can poll for stuck items.\nThat's a state machine with extra steps — literally. Step Functions is AWS's\nmanaged orchestrator for exactly this shape of problem, and it's worth\nknowing precisely when it earns its keep and when it's overkill.
A Step Function is a JSON (or YAML, via the newer workflow studio) state\nmachine defined in Amazon States Language. Each state does one thing —\ninvoke a Lambda, call another AWS service directly via an \"SDK integration,\"\nbranch on a condition, wait, fan out over a list, or hand off to a human\napproval step — and the service itself handles the transitions, retries,\ntimeouts, and error handling between them. Execution history is retained and\nvisualized automatically, so when something fails at 2am you get a diagram\nwith a red X on the exact state that broke, not a pile of CloudWatch Logs\nyou have to stitch together by request ID.
\nThere are two flavors, and picking the right one matters:
\nUsing Standard for a workflow that fires 50,000 times a day doing simple\nAPI-to-API glue work is the single most common way people get an\nunpleasantly large Step Functions bill.
\nMap state runs a step over every item in an\narray — in Distributed Map mode, up to 10,000 concurrent child executions\nreading directly from S3 or a JSON array, without you writing a single\nline of concurrency-control code.If your \"workflow\" is two steps with no meaningful failure branching — call\nAPI A, then call API B with A's result — you don't need a state machine, you\nneed a Lambda function or even just synchronous code in your existing\nservice. The overhead of authoring, deploying, and versioning a state\nmachine definition isn't worth it for something a single function handles in\nten lines. And for very high-throughput, sub-second, simple orchestration\n(think: per-request routing in a hot path), Express workflows can work, but\nyou're often better served by keeping that logic in application code and\nreserving Step Functions for where the auditability and visual execution\nhistory actually pay off.
\nStandard workflows charge per state transition, not per execution — and\ntransitions add up faster than people expect. A Map state iterating over\n1,000 items, each running three sequential states, is 3,000 transitions in\none execution, not one. At $0.025 per 1,000 transitions, that's cheap in\nisolation, but a workflow that fans out over large datasets on a frequent\nschedule can quietly become one of the more expensive things in an account.\nThe fix isn't to avoid Map — it's to use Distributed Map, which counts\nchild workflow executions differently and is built for exactly this\nhigh-fan-out case, plus to actually look at the \"state transitions\" line\nitem in Cost Explorer before a workflow goes from a proof of concept to a\nproduction schedule running every five minutes.
Use ResultSelector and OutputPath inside state definitions to trim what\ngets passed downstream instead of piping entire upstream payloads (including\nthat giant DynamoDB item or Lambda response) through every subsequent state.\nStandard workflow execution history and each state's input/output are\ncapped at 256 KB — passing bloated payloads through unfiltered is the most\ncommon way people hit that limit and get a cryptic\nStates.DataLimitExceeded error on a workflow that's otherwise working\nfine.
CloudFront gives you two different ways to run code at the edge, and they're\nnot tiers of the same thing — they're built for genuinely different jobs,\nwith different runtime models, different latency profiles, and a pricing\ngap wide enough that picking the wrong one for a simple task is a real cost\nmistake, not just a style preference.
\nCloudFront Functions runs a restricted subset of JavaScript in a\nlightweight, purpose-built runtime embedded directly in CloudFront's edge\nlocations — sub-millisecond execution, no cold starts, and it can run on\nviewer request/response events only (the events closest to the end\nuser, before/after CloudFront's cache). It has hard constraints: no network\naccess, no filesystem access, a 10 KB code size limit at the \"basic\" compute\ntier (2 MB on the newer \"advanced\" tier, which raised limits and added\nthings like key-value store access), and a 1 MB max HTTP request/response\nsize to operate on.
\nLambda@Edge runs actual Lambda functions (Node.js or Python, your\nchoice of runtime and package) at CloudFront edge locations, and can run on\nall four CloudFront event types — viewer request, origin request,\norigin response, and viewer response — with network access, larger code\npackages, and access to other AWS services from within the function. It has\nreal cold starts (worse than standard Lambda, since it's replicating across\nedge regions) and materially higher per-invocation and per-GB-second\npricing than CloudFront Functions.
\nThe deciding question isn't \"which is more powerful\" — Lambda@Edge always\nwins that comparison — it's **does this task need origin access or\nfirst-party AWS service calls**. If the answer is no, CloudFront Functions\nis almost always the right choice:
\nindex.html rewrite\npattern static sites need for pretty URLs) — pure string manipulation on\nthe request, no origin needed. CloudFront Functions territory, and\nmeaningfully cheaper and faster than Lambda@Edge for exactly this job.Reach for Lambda@Edge only when the task genuinely needs something\nCloudFront Functions structurally can't do:
\nCloudFront Functions is priced per invocation, at a rate roughly two\norders of magnitude cheaper than Lambda@Edge's per-invocation-plus-duration\npricing, and it runs with no cold start because it's not a full Lambda\nexecution environment being spun up — it's closer to a purpose-built string-\nprocessing VM than a general compute runtime. For a function firing on\nevery single request to a high-traffic distribution (which URL-rewrite and\nheader functions typically do), that pricing gap is not academic — running\na basic redirect rule as Lambda@Edge instead of a CloudFront Function on a\nbusy site can be the difference between a rounding-error cost and a\nnoticeable line item.
\nDefault to CloudFront Functions and only reach for Lambda@Edge when you hit\none of its actual constraints (no network access, code size, event type) —\nnot the other way around. It's easy to reach for Lambda@Edge out of\nfamiliarity with Lambda generally, but for the viewer-request rewrite and\nheader-manipulation jobs that make up most CloudFront edge-compute use\ncases, it's strictly worse on both cost and latency for no functional gain.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "aws", "cloudfront", "spotlight" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-cloudtrail-lake/", "url": "https://jstgtech.com/blog/2026-08-10-cloudtrail-lake/", "title": "Service spotlight: querying years of audit history with CloudTrail Lake", "summary": "How CloudTrail Lake lets you SQL-query months of API activity without standing up Athena and Glue yourself, and where its pricing model changes the calculus.", "content_html": "The standard CloudTrail setup — trail to S3, Athena table on top, manual\npartition projection — works, but it's a pipeline you own: someone has to\nget the Glue table schema right, keep partitions current, and remember it\nexists when an incident finally needs it at 2am. CloudTrail Lake exists to\ndelete that pipeline.
\nCloudTrail Lake is a managed, queryable event store for CloudTrail data.\nYou create an event data store, point it at management events, data\nevents, or both, across one account or an entire AWS Organization, and AWS\ningests, indexes, and retains that data for up to seven years without\nyou touching S3 or Glue. Querying is plain SQL against the event data\nstore through the CloudTrail console or the cloudtrail-data API — no\nAthena table definitions, no partition maintenance, no worrying that a\nschema drift in raw JSON broke your queries.
Under the hood it's still built on the same event structure as classic\nCloudTrail-to-S3, so anything you already know about CloudTrail event\nfields (eventName, sourceIPAddress, userIdentity, requestParameters)\ntransfers directly — the difference is entirely in ingestion and query\nergonomics, not in what's being captured.
DeleteBucket in the\nlast six months\" is a SELECT with a WHERE clause instead of an Athena\ntable setup exercise done under pressure during an active incident. That\ndifference matters most exactly when you have the least patience for\ninfrastructure work.CloudTrail Lake is not a replacement for a trail delivering to S3 if you\nneed that data available to other tooling — a SIEM ingesting raw\nCloudTrail JSON from S3, a security data lake feeding a different query\nengine, or a compliance requirement for immutable S3 object storage with\nObject Lock. Lake's event data store is queryable through its own API and\nconsole, not a general-purpose object store other systems can read from\ndirectly. Most mature setups run both: a trail to S3 for downstream\ntooling and archival, and an event data store for ad hoc SQL investigation.
\nCloudTrail Lake bills per GB ingested, with a choice between two\npricing options at event-data-store creation: one-year extendable\nretention (higher ingestion price, retention extendable up to 7 years,\ndata can be exported) or seven-year retention (lower ingestion price,\nfixed at 7 years, no export). That choice is set per store and isn't\nsomething you casually change later, so decide upfront whether you'll ever\nneed to export the underlying data — if there's any chance you will, pay\nfor the extendable tier even though it costs more per GB, because\nre-ingesting historical data into a differently-configured store isn't an\noption.
\nFor accounts with high API call volume (a busy CI/CD pipeline hitting AWS\nAPIs constantly, or verbose data events on S3/Lambda), ingestion costs can\nadd up fast — model this against your actual CloudTrail event volume\nbefore turning on data events across every account in an Organization,\nrather than discovering the bill after the fact.
\nStart with management events only in the event data store, and add\ndata events (S3 object-level, Lambda invocations) selectively for the\nhandful of buckets or functions where object-level audit trail actually\nmatters — data events are dramatically higher volume than management\nevents, and are the single most common reason a CloudTrail Lake bill comes\nin far higher than expected.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "aws", "cloudtrail", "security", "spotlight" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-kinesis-data-streams/", "url": "https://jstgtech.com/blog/2026-08-10-kinesis-data-streams/", "title": "Service spotlight: Kinesis Data Streams, shards, and when SQS was simpler", "summary": "What shards and fan-out actually buy you over SQS, where consumer scaling gets tricky, and the honest case for not reaching for Kinesis by default.", "content_html": "I've seen more than one team reach for Kinesis Data Streams because\n\"streaming\" sounded like the right word for their use case, then spend\nweeks fighting shard math for a workload that one SQS queue and a Lambda\nwould have handled with a fraction of the operational overhead. Kinesis is\nthe right tool for a specific shape of problem — it's worth being precise\nabout what that shape is before picking it.
\nKinesis Data Streams is an append-only, partitioned log: producers\nwrite records to a stream, the stream is divided into shards, and each\nshard is an ordered sequence that multiple independent consumers can\nread from the same position without deleting records for each other. That\nlast part is the core difference from SQS: an SQS message is deleted once\na consumer processes it, so only one logical consumer group gets each\nmessage, whereas a Kinesis record stays in the stream (default 24 hours,\nextendable to 365 days) and any number of consumer applications can read\nthe full history independently, at their own pace, from their own\ncheckpoint.
\nEach shard supports up to 1 MB/sec or 1,000 records/sec of writes and\n2 MB/sec of reads (5 reads/sec) in standard consumer mode, or up to\n2 MB/sec per shard per consumer with enhanced fan-out, which gives\neach registered consumer a dedicated read throughput instead of sharing\nthe base 2 MB/sec across all of them.
\nIf you have one logical consumer (or a consumer group where each\nmessage should be handled exactly once by exactly one worker), SQS is\nless to operate: no shard count to manage, no manual scaling decision when\nthroughput grows, and its default at-least-once, auto-scaling queue model\njust works without capacity planning. Kinesis in provisioned mode\nrequires you to explicitly resize (split or merge) shards as throughput\nchanges — under-provisioned shards throttle producers with\nProvisionedThroughputExceededException, and over-provisioned shards are\npaying for capacity you don't use. On-demand mode removes manual shard\nmanagement (Kinesis scales automatically based on observed throughput,\nwithin limits), trading that operational burden for a meaningfully higher\nper-stream price than provisioned mode at low-to-moderate volume.
Consumer-side, Kinesis Client Library (KCL) applications carry more\noperational surface than an SQS consumer loop — checkpointing, lease\nmanagement via a DynamoDB table KCL creates for you, and shard rebalancing\non scale-out are all things that can silently misbehave (a consumer\nfalling behind and not alarming on IteratorAge is the classic failure\nmode) in ways a plain SQS ReceiveMessage loop doesn't have to think\nabout at all.
Before reaching for Kinesis, ask whether you actually need **multiple\nindependent consumers replaying the same ordered stream**. If the honest\nanswer is \"no, I have one processing pipeline,\" SQS (or SQS FIFO, if you\nneed ordering within a single consumer group) is very likely the simpler,\ncheaper, lower-maintenance choice — save Kinesis for when the fan-out and\nreplay semantics are the actual requirement, not just the more\nimpressive-sounding service name.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "aws", "kinesis", "streaming", "spotlight" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-route53-health-checks-failover/", "url": "https://jstgtech.com/blog/2026-08-10-route53-health-checks-failover/", "title": "Service spotlight: cheap DNS-level failover with Route 53 health checks", "summary": "How Route 53 health checks and failover routing build automatic DNS failover without a load balancer in front, and where DNS TTLs limit how fast it actually is.", "content_html": "Multi-region failover has a reputation for requiring a global load\nbalancer, cross-region routing infrastructure, and a meaningful\nengineering lift. For a lot of workloads — a static site with a backup\norigin, an API with a standby region, anything where \"route traffic away\nfrom the broken endpoint\" is the whole requirement — Route 53 health\nchecks and failover routing get you most of the way there with nothing\nmore than DNS records.
\nA Route 53 health check polls an endpoint (HTTP, HTTPS, or TCP) on an\ninterval (default 30 seconds, or 10 seconds for the \"fast\" option) from a\ndistributed set of AWS health checker locations worldwide, and marks the\nendpoint healthy or unhealthy based on a configurable failure threshold —\nrequiring multiple consecutive failures across multiple checker locations\nbefore flipping status, which avoids a single flaky network path in one\nregion triggering an unnecessary failover.
\nFailover routing policy then ties DNS answers to that health check\nstatus: you create a primary record pointing at your main endpoint and\nassociate it with the health check, and a secondary record pointing at\na backup endpoint. While the primary's health check passes, Route 53\nanswers DNS queries with the primary record. The moment the health check\nfails, Route 53 stops returning the primary record and starts answering\nwith the secondary — no load balancer, no application-level failover\nlogic, just DNS resolving to a different answer once the health check\ntrips.
\nDNS failover is only as fast as clients actually re-resolve the\nrecord. A record with a 300-second TTL means some fraction of clients —\nand, more unpredictably, any resolver or client that doesn't strictly\nhonor TTL — keep hitting the now-dead primary for up to that TTL after\nfailover triggers, sometimes longer with misbehaving caching resolvers.\nSet a low TTL (30-60 seconds) on records used for failover well before\nyou need it — TTL changes themselves take time to propagate, since\nresolvers that already cached the old TTL keep using it until their\ncurrent cache entry expires, so this isn't something you can fix in the\nmoment of an actual outage.
\nHealth check evaluation itself also isn't instant: with the default\n30-second interval and default failure threshold, detecting an outage and\nflipping DNS can take on the order of a minute or two end to end, which\nis well short of what many RTO targets actually need for a fully\nautomated failover — model that latency into your DR plan rather than\nassuming \"we have failover configured\" means \"we have sub-minute RTO.\"
\nCreate a calculated health check (one that aggregates the status of\nseveral child health checks with AND/OR/NOT logic) when \"healthy\" means\nmore than one endpoint being reachable — e.g., requiring both the API and\nits database dependency's health check to pass before Route 53 considers\nthe primary healthy, instead of failing over on a symptom while the\nactual root cause goes undetected.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "aws", "route53", "reliability", "spotlight" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-s3-intelligent-tiering/", "url": "https://jstgtech.com/blog/2026-08-10-s3-intelligent-tiering/", "title": "Service spotlight: when S3 Intelligent-Tiering beats hand-rolled lifecycle rules", "summary": "How Intelligent-Tiering automates storage-class transitions by access pattern, its monitoring fee, and when a plain lifecycle policy is still cheaper.", "content_html": "S3 lifecycle rules that move objects to Infrequent Access after 30 days\nand Glacier after 90 are a reasonable default when access patterns are\npredictable. They're a bad fit when they're not — and \"predictable access\npattern\" is an assumption worth questioning for a lot of buckets that\nactually hold a mix of hot and cold data with no clean age-based line\nbetween them.
\nS3 Intelligent-Tiering is a storage class that automatically moves\nobjects between access tiers based on observed access patterns\ninstead of a fixed age threshold you define upfront. Objects start in the\nFrequent Access tier; after 30 consecutive days with no access, they\nmove to Infrequent Access automatically; after 90 days with no access,\noptionally to Archive Instant Access; and, if you opt in to the\ndeeper tiers, after 180 days to Archive Access and\nDeep Archive Access — with the key property that if an object in a\ncolder tier is accessed again, it moves back to Frequent Access\nautomatically, and there's no retrieval fee for the Frequent,\nInfrequent, or Archive Instant Access tiers (retrieval from the two\ndeepest archive tiers works like Glacier and does require a restore\nrequest with retrieval time).
\nThis is the core difference from a manual lifecycle policy: a lifecycle\nrule moves objects on a **fixed schedule regardless of whether they're\nstill being accessed**, so an object accessed on day 89 still transitions\nto Glacier on day 90 if that's what the rule says, and pulling it back out\nmeans eating a retrieval cost and (for Glacier-class tiers) a wait.\nIntelligent-Tiering only moves objects that have genuinely gone cold, and\nun-cools them automatically the moment access resumes.
\nIntelligent-Tiering charges a small per-object monthly monitoring and\nautomation fee on top of storage costs, for every object over 128 KB\n(objects smaller than that are charged at Frequent Access rates and\nnever monitored or transitioned, since the monitoring fee would exceed\nany storage savings). For a bucket with millions of small objects, that\nper-object fee adds up fast and can exceed what you'd pay just leaving\neverything in Standard, let alone what a manual lifecycle-to-IA rule would\ncost. And if access patterns genuinely are predictable — logs that\nare always hot for 30 days and never touched again, backups that are\nwritten once and only ever read during a disaster recovery event — a\nfixed lifecycle rule with no monitoring fee at all is strictly cheaper,\nbecause you already know the answer Intelligent-Tiering would spend money\nfiguring out.
\nRun S3 Storage Lens or check the bucket's access patterns via S3 Inventory\nbefore choosing between the two — if you can already describe the access\npattern in one sentence (\"logs go cold after 30 days, always\"), write a\nlifecycle rule. If the honest answer is \"it varies by object and I don't\nknow the split,\" Intelligent-Tiering is worth the monitoring fee to stop\nguessing.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "aws", "s3", "storage", "spotlight" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-secrets-manager-rotation/", "url": "https://jstgtech.com/blog/2026-08-10-secrets-manager-rotation/", "title": "Service spotlight: wiring automatic rotation into Secrets Manager", "summary": "How Secrets Manager rotation actually works end to end with the Lambda rotator pattern for RDS, and the failure modes that leave a secret half-rotated.", "content_html": "Storing a database password in Secrets Manager instead of a .env file is\nthe easy part. The part that actually earns the \"we rotate credentials\"\nline in a security questionnaire is automatic rotation — and that's where\nmost setups I've reviewed stop halfway, with a secret stored but never\nactually rotated because nobody wired up the rotation Lambda.
Secrets Manager stores secrets encrypted with KMS and, for the\nservices that matter most, can rotate them on a schedule automatically\nvia a rotation Lambda function that Secrets Manager invokes. For\nRDS, Aurora, DocumentDB, and Redshift, AWS provides pre-built rotation\nLambda templates (deployed via a SAM app from the Secrets Manager\nconsole or CLI) that implement the standard four-step rotation process\nwithout you writing the logic yourself:
\nAWSPENDINGAWSPENDING to AWSCURRENT, completing the\nrotationThat staged, four-step model exists specifically so a failure partway\nthrough doesn't lock you out: AWSCURRENT only moves to the new password\nafter testSecret confirms it works, so a broken setSecret step leaves\nthe database still accepting the old, still-AWSCURRENT password rather\nthan stranding the app with a password nothing accepts.
GetSecretValue at connection time (ideally through\nthe Secrets Manager RDS/JDBC connector libraries, which handle the\nAWSCURRENT/AWSPENDING transition transparently) instead of having a\npassword baked into config, so a rotation doesn't require an app\ndeploy — as long as the app re-fetches rather than caching the\ncredential for its entire process lifetime.AWSCURRENT, so old connections using the previous user keep working\nuntil they naturally cycle rather than being cut off mid-rotation.The most common failure isn't rotation itself — it's **rotation never\nrunning successfully because the Lambda can't reach the database**. The\nrotation Lambda needs network access to the database (correct VPC\nsubnets, security group rules allowing it in) and the Secrets Manager VPC\nendpoint if the Lambda runs without internet egress; get either wrong and\nrotation fails silently on schedule, over and over, until someone notices\nthe secret's LastRotatedDate hasn't moved in months. Alarm on rotation\nfailures explicitly (CloudWatch metric filter on the rotation Lambda's\nerror logs, or EventBridge on RotationFailed) rather than assuming \"I\nset a rotation schedule\" means it's actually rotating.
The other gap: rotation changes the secret in Secrets Manager and on the\ndatabase, but doesn't retroactively fix every place a credential might be\ncached — a long-lived connection pool that doesn't recycle connections, or\na sidecar that read the secret once at container start and never again,\nkeeps using the old credential until it happens to reconnect. Rotation\nstrategy has to account for how long-lived your actual connections are,\nnot just how often the schedule fires.
\nSet the rotation schedule's testing in a non-production secret first,\nand deliberately break setSecret (point it at a nonexistent user, say)\nto confirm the four-step staging really does leave AWSCURRENT untouched\non failure before you trust it against a production database credential —\nverifying the failure mode is safe is worth the ten minutes it takes.
I still see security groups with port 22 open to a bastion host's IP range,\na bastion host someone has to patch, and an SSH key rotation process nobody\nactually follows. Session Manager has been GA for years and solves this\ncompletely, and yet the bastion pattern persists mostly out of habit — worth\na clear-eyed look at what actually changes when you switch.
\nSession Manager is a feature of AWS Systems Manager that opens an\ninteractive shell session to an EC2 instance (or on-prem/other-cloud server\nrunning the SSM agent) entirely over the agent's outbound HTTPS connection\nto the SSM service — no inbound port needs to be open on the instance at\nall, not even 22. Authentication and authorization happen through IAM:\nif your IAM identity has ssm:StartSession permission (typically scoped by\nresource tag) and the instance has SSM's managed policy on its instance\nrole, you get a shell. No SSH key to distribute, no bastion host to\nmaintain, no security group rule to remember to remove later.
Every session is logged: session start/end, the identity that connected,\nand — if you enable it — the full session transcript to CloudWatch Logs\nor S3, plus KMS encryption of session data in transit. That's a\nmeaningfully better audit story than SSH access, where \"who ran what\ncommand on this box\" usually means reconstructing it from shell history\nfiles that a user can edit or delete.
\ncom.amazonaws.region.ssm\nplus the two related endpoints), so instances with zero internet egress\nare still reachable for administration.aws ssm start-session\n --document-name AWS-StartPortForwardingSession tunnels a local port to a\nremote one (handy for reaching an RDS instance in a private subnet from a\nlaptop) without opening the database to the internet or standing up a\njump host at all.The SSM agent has to be installed, running, and able to reach the SSM\nservice endpoints — most current AMIs (Amazon Linux 2023, recent Ubuntu/\nWindows AMIs) ship with it preinstalled, but older custom AMIs and\nminimal/hardened base images often don't, and \"Session Manager isn't\nworking\" on those turns out to be a missing or outdated agent, not an IAM\nproblem. The instance also needs outbound HTTPS reachability to the SSM\nendpoints — either a NAT gateway/instance, or the three SSM-related VPC\ninterface endpoints for fully private subnets — and forgetting the VPC\nendpoints on a private-subnet instance is the most common \"it works in one\nVPC and not another\" support ticket.
\nThe other gap: Session Manager solves interactive shell access, but it's\nnot a substitute for scoped, least-privilege instance role permissions.\nSomeone with ssm:StartSession on an instance whose instance role has\nbroad S3 or IAM permissions still inherits that instance's blast radius once\nthey're in the shell — Session Manager changes how you get access, it\ndoesn't change what that access can do once you're there.
Turn on session logging to CloudWatch Logs and set a retention/alerting\npolicy on it from day one — the audit trail is the single biggest advantage\nover SSH, and it's only useful if someone's actually watching or alarming on\nit, not just accumulating in a log group nobody queries until an incident\nforces the question.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "aws", "ssm", "security", "spotlight" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-well-architected-tool/", "url": "https://jstgtech.com/blog/2026-08-10-well-architected-tool/", "title": "Service spotlight: running an AWS Well-Architected Tool review", "summary": "What the Well-Architected Tool actually surfaces when you run a workload through it, where its generic questions fall short, and how to act on the output.", "content_html": "\"We should do a Well-Architected review\" tends to get said in the same\ntone as \"we should write more tests\" — a good idea everyone agrees with\nand nobody schedules. Having actually run several of these against real\nproduction workloads, the tool is more useful than its reputation as a\ncompliance checkbox exercise suggests, provided you go in knowing what it\nwill and won't tell you.
\nThe AWS Well-Architected Tool is a free console tool that walks a\ndefined workload (you register one per application or system you want\nreviewed) through a structured questionnaire organized by the **six\npillars**: Operational Excellence, Security, Reliability, Performance\nEfficiency, Cost Optimization, and Sustainability. Each pillar has a set\nof questions (\"How do you manage identity and permissions for people and\nmachines?\"), each question has a list of best-practice choices you\nselect as implemented or not, and unchosen best practices become flagged\nrisk items — categorized as high or medium risk — in a summary report\nper pillar.
\nThe output is a milestone: a snapshot of your answers and risk items\nat a point in time, which you can re-run later and diff against to show\nwhether risk actually went down after remediation work, not just that a\nreview happened once.
\nThe questionnaire is generic by design, which means its questions are\nnecessarily abstracted away from your actual architecture — it will ask\nwhether you have a documented incident response runbook, but it can't\ntell you whether your specific runbook is any good, or whether your\nDR RTO target is realistic for your actual failure modes. It's a\nstructured prompt for a conversation your team needs to have, not a\nsubstitute for that conversation, and treating it as a checkbox exercise\nwhere you tick \"implemented\" without actually verifying the practice is\nin place defeats the entire point — self-reported answers are only as\nhonest as the person answering them.
\nIt also doesn't automatically inspect your account — answering the\nquestionnaire accurately still requires someone to actually go look at\nwhat's configured, which is real work. (AWS's separate Trusted\nAdvisor and Well-Architected Tool's own automated checks\nintegration can pull some findings in automatically for a subset of best\npractices, narrowing but not eliminating that manual-verification gap.)
\nDon't try to review an entire application's full six-pillar surface in\none sitting — split the review across a couple of working sessions per\npillar with the people who actually own that area (security engineer for\nthe Security pillar, whoever owns the on-call rotation for Reliability),\nand treat the output high-risk items as backlog tickets with owners, not\na report that goes in a drawer once the review is done.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "aws", "wellarchitected", "architecture", "spotlight" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-arista-velocloud-cve-2026-16812/", "url": "https://jstgtech.com/blog/2026-08-10-arista-velocloud-cve-2026-16812/", "title": "CVE-2026-16812: a CVSS 10 hole in your SD-WAN control plane", "summary": "An unauthenticated command injection in Arista VeloCloud Orchestrator On-Prem lets attackers pivot from one exposed console to every edge device it manages.", "content_html": "Most weeks the scariest CVE is scary because of what it's attached to — a\ndatabase, a CI/CD server, a pile of customer data. This week it's scarier\nbecause of what it's in front of: the single management plane for an\nentire SD-WAN fleet. CVE-2026-16812, an unauthenticated OS command\ninjection in Arista's VeloCloud Orchestrator (VCO) On-Prem, scores a\nperfect CVSS 10.0, is under active exploitation, and CISA added it to the\nKnown Exploited Vulnerabilities catalog with a federal remediation deadline\nof July 30 (CISA). If you run any branch offices, retail sites, or hybrid-cloud links over\nVeloCloud, this is the one to stop and read carefully.
\nVCO is the web console that operators use to configure and monitor every\nVeloCloud Edge device in a deployment — think of it as the \"cloud console\"\nfor your own private SD-WAN. According to Arista's advisory, the flaw lets\n\"remote attackers access privileged functionality that was intended only\nfor internal use and should not be remotely accessible\" (BleepingComputer). In plain terms: some internal-only code path that shells out to the\nunderlying OS is reachable from the external web interface without\nauthentication, and attacker-controlled input reaches that shell call\nunsanitized. No credentials, no prior access, no user interaction — just\nnetwork reachability to the VCO web UI is enough to get arbitrary command\nexecution as whatever user the orchestrator process runs as.
\nThe uncomfortable detail from The Register's coverage is that the exposure\nis structural: the orchestrator is \"exposed by default, with no\nconfiguration capable of removing that exposure entirely\" (The Register). This isn't a case of someone forgetting to put the admin panel behind a\nVPN — On-Prem VCO's own design assumes some surface has to be internet-\nreachable for edge devices to phone home to it, and that's the same surface\nthe vulnerable code path sits on. Arista's Hosted and Dedicated (SaaS)\ndeployments were already patched before the advisory went public, which\nnarrows this specifically to organizations running their own VCO\ninstance rather than using Arista's managed service.
\nThis is where CVE-2026-16812 earns the CVSS 10. Compromising VCO doesn't\njust hand over one server — it hands over the control plane for every\nVeloCloud Edge device the orchestrator manages: branch routers, site\nconfigurations, routing policy, and the tunnels those edges use to reach\neach other and the cloud. Arista's own guidance says exploitation \"may\ncompromise the confidentiality, integrity, and availability of the\norchestrator and data managed by the orchestrator,\" and researchers have\nflagged the realistic follow-on as attackers using orchestrator access to\npush malicious configuration to managed edges, effectively turning a single\nweb app bug into a foothold across every site in the WAN. If your VCO\nmanages edges that terminate into a VPC or a colo where your workloads\nlive, that's the pivot path from \"someone popped our SD-WAN console\" to\n\"someone is on our network.\"
\nCISA has already observed active exploitation, and researchers have\npublished three IP addresses seen scanning and delivering payloads:\n8.19.75.217, 206.72.242.124, and 206.72.242.162. Worth feeding those\ninto your firewall/IDS blocklists today regardless of your patch status —\nthey're a known-bad signal, not a mitigation on their own.
Arista has shipped fixed builds for every affected On-Prem branch:
\nPatch first, but don't stop there. Because the flaw grants unauthenticated\nRCE and has apparently been exploited since before public disclosure,\nArista's own advisory goes further than \"update and move on\" — it\nrecommends rotating credentials, validating that managed edge devices\nhaven't had unauthorized configuration changes pushed to them, and treating\na confirmed-compromised instance as a candidate for restore-from-clean\nrather than trusting an in-place patch. That's a strong signal from the\nvendor that patching alone may not be sufficient remediation if you can't\nrule out prior compromise — check orchestrator and edge audit logs for\nactivity around the known-bad IPs above before you consider this closed.
\nIf patching isn't immediate, restrict the VCO web interface to trusted\nmanagement networks now. It won't fully close the hole given how the\nproduct's exposure is architected, but cutting off casual internet\nscanning buys time, and it's the same \"assume it's reachable, so fence it\"\nposture you'd want for any management plane with this much downstream\nreach.
\nThe recurring theme this year is that orchestration and management planes\n— CI/CD servers, RMM consoles, and now SD-WAN orchestrators — are\nincreasingly the preferred target, precisely because compromising one\nsystem fans out into control over everything it manages. If you run\non-prem management software for infrastructure with real blast radius, the\nquestion worth asking isn't just \"is it patched\" but \"what does an\nattacker get if this one box falls\" — and whether that answer is something\nyou're comfortable with by design, not just by patch level.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "security", "aws", "networking" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-github-actions-supply-chain/", "url": "https://jstgtech.com/blog/2026-08-10-github-actions-supply-chain/", "title": "CVE-2025-30066: The GitHub Action Tag You Trusted Lied", "summary": "A compromised maintainer token let attackers rewrite tj-actions/changed-files version tags, dumping CI/CD secrets from thousands of repos into public build logs.", "content_html": "Every workflow file with a line like uses: tj-actions/changed-files@v45\nis making a promise to itself that a tag won't change out from under it.\nIn March 2025 that promise broke for real: attackers rewrote the tags on\ntj-actions/changed-files, a GitHub Action used in over 23,000 repositories,\nto point at a malicious commit that dumped CI/CD secrets straight into\npublic workflow logs — tracked as CVE-2025-30066 (CISA). It's one of the cleanest real-world illustrations of why \"pin a version\"\nand \"pin a commit\" are not the same security control, and it's directly\nrelevant if your CI — like this site's — runs on GitHub Actions.
tj-actions/changed-files is a popular action that reports which files\nchanged in a PR or push, and plenty of pipelines gate steps on its output.\nOn March 14, 2025, its maintainers discovered that a large number of the\nproject's version tags had been silently repointed to a commit that never\nwent through review: it injected a Node.js payload that scanned GitHub\nRunner memory for credentials — cloud access keys, PATs, npm tokens, private\nRSA keys — and printed them, base64-encoded, into the workflow run log\n(Wiz).
The entry point traces back further, and it's the more interesting part of\nthe story. Wiz's investigation found the compromise was likely cascading:\nthree days earlier, on March 11, attackers had already hijacked the v1\ntag of a different action, reviewdog/action-setup, and pointed it at\nmalicious code of their own (CVE-2025-30154) (StepSecurity). tj-actions/eslint-changed-files happened to depend on\nreviewdog/action-setup, and the maintainers' own bot ran with a GitHub\npersonal access token — a token that Wiz believes was harvested through\nthat same compromised dependency and then used to push the malicious commit\nand rewrite the changed-files tags (Wiz). One stolen credential in one small action rippled into a second,\nmuch more widely used one within days.
The structural flaw both incidents share is that a tag like @v1 or @v45\nis just a mutable pointer, not a content hash. GitHub lets any tag be force-\npushed to a new commit at any time by anyone with write access — there's no\ncryptographic binding between the ref your workflow trusts and the code\nthat actually runs when the runner checks it out. Reviewing an action once\nand pinning @v4 doesn't protect you from a maintainer account (or its\nautomation token) getting popped six months later.
The initial estimate — every one of the 23,000+ repos using\ntj-actions/changed-files — made headlines fast, but the real exposure\nwindow was narrower and still meaningfully bad. The malicious code was live\nbetween roughly March 12, 00:00 UTC and March 15, 12:00 UTC; anyone who ran\nthe action against a public repo during that window had whatever secrets\nwere in scope for the job printed straight into a log anyone could read,\nbefore GitHub could pull them down (GitHub Advisory Database). On the upstream reviewdog side, deeper analysis found the confirmed\nsecret-leak count was smaller than the initial panic suggested — around 218\nrepositories actually had secrets exposed in logs, not tens of thousands\n(BleepingComputer) — but \"smaller than feared\" still means real cloud credentials and\ntokens sitting in public logs for anyone who scraped them in time. GitHub\ntemporarily pulled the tj-actions/changed-files repository entirely while\nthe malicious commit was reverted and tags restored, which is its own\nsignal of how seriously the platform treated it.
The fix that actually closes this hole is boring and mechanical: pin every\nthird-party Action to a full 40-character commit SHA, not a tag —\nuses: tj-actions/changed-files@a1b2c3... instead of @v45. A SHA can't\nbe silently repointed the way a tag can, so a compromised maintainer token\ncan rewrite tags all day and your pinned workflow still runs the commit you\nreviewed. Pinning by SHA does mean you lose automatic updates, which is why\nyou pair it with Dependabot or Renovate — both can open PRs that\nbump the pinned SHA (with the new tag as a comment for readability) so\nupgrades stay a reviewed diff instead of an invisible tag move. On the org\nside, GitHub lets you restrict which Actions are allowed to run at all\n(allow-list specific actions or require them to be from verified creators),\nwhich is worth turning on for anything beyond a personal repo. If you ran\nan affected action during the exposure window, don't stop at patching: pull\nthe workflow run logs, search them for anything that looks like a leaked\ncredential, and rotate every secret that was in scope for those jobs —\nassume exposure rather than hoping the log got pulled in time.
A third-party GitHub Action is a dependency with write access to your CI\nenvironment and, by extension, whatever your CI can reach — cloud\ncredentials, deploy keys, package registry tokens. Marketplace popularity\nand a green checkmark on the README aren't a security review; they're a\npopularity contest. This site's own CI runs entirely on GitHub Actions, and\nthe practical takeaway is the same one that applies to any npm install:\npin to something immutable, let a bot handle the update diffs, and treat\nevery action in your workflow file as code you're choosing to run with your\nsecrets — because that's exactly what it is.
If you manage remote access for a living, Ivanti Connect Secure has probably\ncost you a weekend at some point in the last three years. The latest entry is\nCVE-2025-0282, an unauthenticated stack-based buffer overflow in Connect\nSecure, Policy Secure, and Neurons for ZTA gateways that Mandiant caught being\nexploited as a zero-day starting in mid-December 2024, weeks before Ivanti\nshipped a fix (Google Cloud / Mandiant). CISA added it to the Known Exploited Vulnerabilities\ncatalog and gave federal agencies until January 15 to patch (CISA KEV). It's the third\nmajor Ivanti edge-device RCE chain in two years, and that pattern is the\nactual story here.
\nCVE-2025-0282 is a classic memory-safety bug: a stack-based buffer overflow\n(CWE-121) in a web-facing component of Connect Secure that a remote,\nunauthenticated attacker can trigger with a crafted request, landing arbitrary\ncode execution on the appliance with no credentials and no user interaction\n(Rapid7). It carries a CVSS base score of 9.0 — full compromise, network\nattack vector, no privileges required.
\nThis isn't Ivanti's first appliance-RCE rodeo. A year earlier it was\nCVE-2023-46805 (an auth bypass) chained with CVE-2024-21887 (a command\ninjection) to get the same outcome on the same product line (CISA). The\nunderlying story repeats: Connect Secure is a Linux-based appliance running a\nweb application stack that's directly exposed to the internet by design —\nthat's the whole point of a VPN gateway — and a memory-corruption or\nauth-logic bug in that stack turns \"reachable on 443\" straight into root on\nthe box. There's no perimeter behind the perimeter device.
\nMandiant's incident response found a genuinely elaborate malware ecosystem\ndropped on compromised appliances, tracked as the SPAWN family: SPAWNANT\n(the installer/persistence mechanism), SPAWNMOLE (a tunneler for pivoting\ninto the internal network), SPAWNSNAIL (an SSH backdoor), and\nSPAWNSLOTH (a log-tampering tool to blind forensic analysis) — plus two\nnewly observed tools, the PHASEJAM dropper and DRYHOOK credential\nharvester (Google Cloud / Mandiant). Most notably, SPAWNANT was built specifically\nto tamper with Ivanti's own Integrity Checker Tool manifest, so the appliance\nwould report clean to the exact tool defenders were told to trust.
\nMandiant attributes the activity, with medium confidence, to UNC5337 —\nbelieved to be part of UNC5221, the same China-nexus espionage cluster behind\nthe 2023/2024 Ivanti chain — going after VPN gateways specifically because\nthey sit at the network edge with credentials and session state flowing\nthrough them and, historically, no EDR agent watching what runs on them\n(The Record). At the time of disclosure, tens of thousands of Connect\nSecure instances were sitting exposed to the internet, and exploitation had\nalready been underway for roughly a month before the public advisory\n(CyberSecurityNews). And the pattern didn't stop there: three months later, Ivanti\ndisclosed CVE-2025-22457, another unauthenticated stack-based buffer\noverflow in the same product line — initially misjudged as a low-severity\ndenial-of-service bug until Mandiant showed it was remotely exploitable —\nand again observed exploited in the wild by a suspected China-nexus actor\nbefore agencies could patch (Google Cloud / Mandiant).
\nPatched builds for CVE-2025-0282 have been available since January 8, 2025:\nConnect Secure 22.7R2.5, Policy Secure 22.7R1.2, and Neurons for ZTA\ngateways 22.7R2.3 (Ivanti advisory). If you're still behind those\nversions, that's the first move, no exceptions.
\nBut given that SPAWNANT was purpose-built to falsify the Integrity Checker\nTool's output, a clean ICT scan on an unpatched or recently patched device\nisn't proof of a clean appliance. Run both the internal and external ICT, but\ntreat it as one signal among several, not a verdict — cross-reference against\nIvanti's published indicators of compromise, check for unexpected outbound\ntunnels or SSH listeners, and if you have any reason to believe a device was\nexposed during the exploitation window, the safer path is a factory reset and\nclean rebuild from a patched image rather than trusting an in-place upgrade\nto have removed a persistence mechanism designed to survive exactly that.\nRotate every credential and certificate that ever transited the appliance —\nVPN gateways see user passwords, session tokens, and often service-account\nsecrets, all of which should be considered burned if compromise can't be\nruled out.
\nThree major RCE chains on the same Ivanti product line in under two years,\neach exploited as a zero-day before a patch existed, each attributed to\nstate-nexus actors going straight for the edge — this isn't bad luck, it's a\ntarget selection. Edge appliances are internet-facing by requirement, run\nvendor firmware most security teams can't instrument the way they'd\ninstrument a server, and sit exactly where credentials and network access\nconverge. If you operate any VPN gateway, firewall, or SSL-VPN appliance —\nIvanti or otherwise — the operational question isn't \"are we patched\" as a\none-time checkbox, it's whether you have a standing plan for zero-day\nexploitation on a box you can't put an agent on: network segmentation around\nthe appliance, egress monitoring for exactly the kind of tunneling SPAWNMOLE\ndoes, and a rebuild-from-clean playbook you've actually rehearsed before you\nneed it at 2am.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "security", "vpn", "networking", "cve" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-java-deserialization-post-log4shell/", "url": "https://jstgtech.com/blog/2026-08-10-java-deserialization-post-log4shell/", "title": "Java deserialization after Log4Shell: CVE-2023-46604", "summary": "Apache ActiveMQ's unauthenticated RCE shows Java deserialization bugs never went away after Log4Shell — only the exploitation playbook got faster.", "content_html": "Log4Shell (CVE-2021-44228) made \"Java deserialization\" a household phrase for\na few terrifying weeks in December 2021, but it wasn't a one-off. Less than\ntwo years later, CVE-2023-46604, an unauthenticated remote code execution\nbug in Apache ActiveMQ's OpenWire protocol, gave attackers the same\noutcome — arbitrary command execution with zero credentials — and it wasn't\na lookup-and-log JNDI trick this time, it was textbook insecure\ndeserialization in a message broker that sits on the network path of a huge\nnumber of Java shops. CISA added it to the Known Exploited Vulnerabilities\ncatalog within days, and ransomware crews had it weaponized before most\ndefenders had patched (Rapid7). If you assumed Log4Shell was the last time a Java\nserialization bug would take down a fleet of production systems overnight,\nthis one is worth sitting with.
\nActiveMQ's OpenWire is the binary wire protocol brokers and clients use to\ntalk to each other, and by default it's listening on port 61616 with no\nauthentication required to establish a connection. Inside that protocol,\none command type — ExceptionResponse — carries a class name and a message\nstring so a broker can tell a client \"here's the exception that happened.\"\nThe marshalling code that unpacks it, BaseDataStreamMarshaller.createThrowable,\ntakes that attacker-supplied class name off the wire and instantiates it\ndirectly, passing the attacker-supplied message string into the constructor,\nwithout first checking that the class is actually a Throwable (Rapid7).
That's the whole bug. Anyone who can open a TCP connection to the OpenWire\nport can send a crafted EXCEPTION_RESPONSE packet naming any class on the\nserver's classpath — including Spring's ClassPathXmlApplicationContext,\nwhich happily fetches and executes an XML bean definition from an\nattacker-controlled URL. Point that at a remote XML file that defines a\nProcessBuilder-backed bean and the broker runs your shell command. It's\nthe same family as the Commons Collections gadget chains that made Java\ndeserialization famous a decade ago — take a class that legitimately does\nsomething dangerous when instantiated or invoked, and abuse a deserializer\nthat doesn't discriminate about what it's allowed to construct. The fix\nApache shipped adds exactly the type check that should have been there from\nthe start: reject any \"exception\" class name that isn't actually a\nThrowable before instantiating it (Rapid7).
CVE-2023-46604 scores CVSS 9.8, and researchers rated exploitation\ncomplexity as trivial — no auth, no user interaction, one crafted packet\n(Huntress). The result is arbitrary command execution as whatever OS user\nruns the broker process, and in the wild that turned into exactly what you'd\nexpect once a reliable pre-auth RCE with a public PoC exists: a race between\nopportunistic and targeted actors. Rapid7 tracked exploitation attributed to\nHelloKitty ransomware starting within days of disclosure (Rapid7), Trend Micro and Sekoia both documented the Kinsing\ncryptomining botnet using it to drop miners and rootkits on Linux hosts\n(Trend Micro, Sekoia), and SOC Prime and others tracked TellYouThePass\nransomware riding the same bug (SOCRadar). Message brokers tend to sit deep in the\narchitecture — application servers, integration layers, IoT backends all\ntalk to them — so a broker compromise isn't an edge-of-network incident,\nit's a foothold with a direct line to whatever internal services trust that\nbroker's traffic.
\nApache patched the marshalling logic and shipped fixed releases across every\nsupported branch: 5.15.16, 5.16.7, 5.17.6, and 5.18.3, with 6.0.0 also\ncarrying the fix (Apache ActiveMQ). If you're still running an\nunpatched broker, upgrading is non-negotiable — this isn't a \"add\nauthentication in front of it\" situation, because the flaw is in how the\nbroker parses its own wire protocol before any application-level auth\napplies. Restricting network access to port 61616 to only trusted broker\nand client hosts is a reasonable stopgap while you schedule the upgrade, but\ngiven how quickly this was weaponized, treat any internet-facing or\nbroadly-reachable OpenWire port as already assumed-compromised and hunt for\nfollow-on cryptominer or webshell activity, not just confirm the patch\napplied.
\nWhat's actually changed since Log4Shell is the plumbing around detection:\nWAF and IDS vendors now ship signatures for OpenWire-style deserialization\npayloads within hours of a PoC landing, CISA's KEV catalog gives defenders a\nauthoritative \"this is being exploited right now\" signal instead of relying\non vendor advisories alone, and software composition analysis tools flag\nvulnerable broker and library versions in CI before they ever ship. On the\nJDK side, JEP 415's context-specific deserialization filters (Java 17+) let\nyou scope an allowlist to a specific ObjectInputStream instead of one\nJVM-wide filter that's either too loose or breaks half your app (Baeldung) — a real improvement over JEP 290's blunt instrument.
What hasn't changed is the underlying pattern: a component trusts a class\nname or a byte stream from the network more than it should, and something\nreachable on the classpath turns that trust into code execution. A 2024\nNDSS study found over 3,600 GitHub Java projects still carrying known\ndeserialization vulnerabilities (Java Code Geeks) — the detection tooling\nis faster, but it's still catching the same class of bug, not preventing\nit from being written. If your Java services still do implicit\nObjectInputStream deserialization or unauthenticated binary protocol\nparsing anywhere on the network path, Log4Shell and ActiveMQ are both\ntelling you the same thing: the fix belongs in the code that trusts the\nbytes, not in the WAF rule that flags them after the fact.
You don't need a zero-day to take over a Kubernetes cluster. You need an\nAPI server that answers on the public internet and an anonymous-auth\nflag nobody turned off. Aqua Nautilus researchers spent three months\nscanning for exactly that combination and found it everywhere: hundreds of\norganizations with clusters already under active attack, most of it\nopportunistic cryptomining riding in through anonymous access that had\nsomehow been granted real privileges (CSO Online, Aqua Nautilus). This isn't a rare edge case — Shadowserver's internet-wide scans have\nrepeatedly found north of 380,000 Kubernetes API servers responding\npublicly, the large majority in the US and Western Europe (Shadowserver, The Register). Exposure alone doesn't mean compromise, but it's the precondition every\ncampaign in this space depends on, and the misconfigurations that turn\n\"reachable\" into \"owned\" are boringly common.
Kubernetes has two places where \"no credentials\" quietly becomes \"some\ncredentials.\" The first is the API server and kubelet's anonymous-auth\nsetting, which historically defaulted to enabled and maps unauthenticated\nrequests to the system:anonymous user (or the system:unauthenticated\ngroup). By itself that's supposed to be harmless — anonymous requests\nshould have no RBAC permissions bound to them. In practice, Aqua's kubelet\nresearch found operators binding that anonymous identity to real roles,\nsometimes admin-level ones, usually as a shortcut to get some internal\ntool or health check working without wiring up proper auth (Aqua Security). Once that binding exists, anyone who can reach the API server or the\nkubelet's HTTPS port (10250) has exactly the access that role grants — no\nlogin required. Aqua's kube-hunter tool has a dedicated check for this\npattern precisely because it recurs so often across audited clusters\n(Aqua Security / kube-hunter).
The second gap is the default service account. Every pod gets one mounted\nautomatically unless a manifest explicitly sets\nautomountServiceAccountToken: false, and that token grants whatever RBAC\nrole is bound to the account — which in a lot of real clusters is broader\nthan anyone intended, because default service accounts accumulate\npermissions over time as people bind roles the fast way instead of the\ncorrect way. Layer that onto an internet-reachable control plane —\nsometimes because of a misconfigured cloud load balancer, sometimes\nbecause someone ran kubectl proxy --address=0.0.0.0 --accept-hosts='.*'\non a bastion and forgot about it, a specific misconfiguration Aqua called\nout by name in its research — and a single unauthenticated HTTP request is\nenough to start probing.
The API server isn't just another service — it's the thing that can\nschedule code onto every node in the cluster. Anonymous or default-account\naccess that includes pod-create permission lets an attacker launch a pod\nwith a hostPath mount or privileged: true, which is a direct path off\nthe container and onto the underlying node. From the node, the attacker\ncan read the kubelet's credentials and every service account token\nscheduled there, pivoting sideways into other namespaces the original\naccess point never touched. In cloud-hosted clusters, pods often carry\nworkload-identity credentials (IRSA on EKS, Workload Identity on GKE) or\ncan reach the instance metadata service directly, so cluster compromise\nroutinely becomes cloud-account compromise. This is also the exact class\nof risk that made CVE-2018-1002105 so severe a few years back — a bug that\nlet unauthenticated requests reach the API server's proxy layer and come\nout the other side with cluster-admin, a reminder that \"you're talking to\nthe API server\" and \"you have privileged access\" are one and the same\nthreat model even without a misconfiguration involved (Tenable). In practice, most of what Aqua and others observe on these exposed\nclusters is quieter than that: TeamTNT- and Kinsing-style campaigns that\ndrop XMRig miners across every node they can reach, plus at least one\ndocumented \"RBAC buster\" campaign that used the access specifically to\nplant a persistent backdoor role rather than just mine coins (Tigera, CSO Online).
--anonymous-auth=false\nand route real authentication through --client-ca-file or a token\nwebhook; never bind system:anonymous or system:unauthenticated to\nany ClusterRole, including \"harmless-looking\" ones.default\nservice accounts — Kubernetes' own docs have a good-practices guide for\nthis and it's worth running against every namespace, not just the\nobviously sensitive ones (Kubernetes docs).automountServiceAccountToken: false on any pod that doesn't\nactually call the Kubernetes API, so a compromised pod doesn't\nautomatically hand over a credential.None of this requires a novel exploit — every campaign cited here rode in\non defaults nobody explicitly locked down. Kubernetes was built to be\nflexible for cluster operators first and secure-by-default second, and\nthat tradeoff means the burden of hardening anonymous access, RBAC\nbindings, and network exposure sits entirely on whoever stood the cluster\nup. Attackers aren't finding your cluster through cleverness; they're\nrunning the same internet-wide scans Shadowserver publishes for free and\nchecking whether you did the hardening step or skipped it. Assume they've\nalready scanned you, and go verify which answer they got.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "security", "kubernetes", "rbac", "cloud" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-leaked-credentials-public-repos/", "url": "https://jstgtech.com/blog/2026-08-10-leaked-credentials-public-repos/", "title": "Leaked credentials in public repos get used in minutes", "summary": "GitGuardian logged 28.65M secrets on public GitHub in 2025, and researchers have watched leaked AWS keys get abused in under five minutes.", "content_html": "GitGuardian's 2026 State of Secrets Sprawl report counted 28.65 million new\nhardcoded secrets pushed to public GitHub commits in 2025 alone — a 34%\njump year over year and the largest single-year increase they've recorded\n(GitGuardian).\nThat's not a backlog of old mistakes; it's this year's output. And the\nsame report found that 64% of secrets that leaked back in 2022 are *still\nvalid* today (GitGuardian).\nMeanwhile Unit 42's tracking of the EleKtra-Leak campaign found attackers\ndetecting and using exposed AWS IAM credentials within roughly five minutes\nof them landing on GitHub (Unit 42).\nCommitting a secret to a public repo isn't a \"someone might find this\neventually\" risk. It's closer to broadcasting it live.
\nAlmost none of this is malicious. It's .env files added to a repo before\nanyone wrote a .gitignore entry for them, database URLs hardcoded into a\nconfig \"just for local testing,\" service-account JSON dropped into a test\nfixture, or a CI job that echoes an environment variable into build logs\nthat are themselves public. GitGuardian's report also flags AI-assisted\ncoding as a growing contributor — commits with AI-generated code leak\nsecrets at roughly double the baseline rate, and secrets tied to AI\nservices specifically were up 81% year over year (GitGuardian),\nwhich tracks: an assistant that's never seen your .gitignore will happily\npaste a working API key straight into a code sample.
The part engineers consistently underestimate is that deleting the secret\nin a follow-up commit doesn't delete it from the repo. Git preserves\nhistory — the credential is still sitting in an earlier commit object,\nreachable by anyone who clones the repo or even just browses the commit\nlog on GitHub. Force-pushing a rewritten history to your own branch\ndoesn't help either if the repo was ever public even briefly: forks, local\nclones, and GitHub's own caches can retain the blob indefinitely. The only\nthing that actually neutralizes a leaked secret is invalidating the\ncredential itself — history rewrites are cleanup, not remediation.
\nThis is the part that should change how you triage. Automated scanners\nwatch GitHub's public event stream continuously, and multiple independent\nresearch efforts have clocked exploitation in the single-digit minutes:\nUnit 42 measured attackers weaponizing exposed IAM keys from the\nEleKtra-Leak campaign in about five minutes (Unit 42),\nand independent write-ups of live incidents describe leaked AWS keys being\ngrabbed and turned into GPU cryptomining fleets within 4 to 11 minutes of\nthe push, run across every region the credential could reach\n(TechRadar).\nThe economics are simple: spinning up GPU instances for Monero mining\ncosts the attacker nothing (it's your bill) and the scanning infrastructure\nis fully automated, so there's no human in the loop slowing things down.\nThere is no \"we'll rotate it during the next sprint\" window here — by the\ntime a human notices the commit, the credential has often already been\nused.
\nThe controls that matter, roughly in order of leverage:
\ngitleaks or trufflehog run\nlocally before a commit is even made, catching secrets in formats GitHub\ndoesn't recognize (internal API keys, custom tokens) and catching them\nbefore they touch history at all, which is strictly better than catching\nthem after.git filter-branch or a force-push as the first response — rotate\nthe credential at the source first. A scrubbed history with a still-valid\nkey behind it protects nobody; an invalidated key with a messy history is\na Tuesday.Prevention controls — push protection, pre-commit hooks, .gitignore\ndiscipline — are worth having, but GitGuardian's own numbers show they\naren't closing the gap: secrets leaked have grown 152% since 2021 against\na 98% growth in GitHub's developer base (GitGuardian),\nmeaning the leak rate is outpacing the population producing the leaks.\nSome secret will eventually get past whatever scanner you've configured,\nbecause prevention is a filter and filters have gaps by definition. What\nactually determines whether that leak becomes an incident report or a\nnon-event is how fast you can rotate the credential and how little damage\nit can do in the window before you notice. That's why the real investment\nisn't just \"stop secrets from leaking\" — it's \"assume one will leak, and\nmake sure it's short-lived, narrowly scoped, and rotated before a bot ever\ngets a chance to use it.\"
In May 2023 the Clop ransomware group quietly started exploiting an\nunpatched SQL injection in Progress Software's MOVEit Transfer — days\nbefore anyone outside the attackers knew it existed. By the time the dust\nsettled, more than 2,700 organizations and upward of 90 million individuals\nhad data stolen through CVE-2023-34362, making it one of the largest\nsingle-vulnerability breaches on record (CISA, TechCrunch). It wasn't an isolated incident — it was the clearest example yet of a\npattern that keeps repeating against managed file transfer (MFT) software:\nfind one flaw in an internet-facing appliance built to move sensitive data,\nand turn it into hundreds of breaches in a matter of days.
\nMOVEit Transfer's vulnerability was a textbook SQL injection: an\nunauthenticated attacker could send crafted input to the web application\nand manipulate backend SQL queries against MySQL, Microsoft SQL Server, or\nAzure SQL, escalating from unauthorized database access to remote code\nexecution (Rapid7). Clop used it to drop a custom ASP.NET web shell — dubbed LEMURLOOT,\ntypically written to disk as human2.aspx to blend in with MOVEit's\nlegitimate human.aspx — giving them a durable, authenticated-looking\nfoothold for pulling files straight out of the transfer database (Mandiant). Forensic teams found evidence attackers had mapped the database schema\nin advance, meaning this wasn't smash-and-grab improvisation — it was a\nprepared exploit chain, sat on for weeks, then fired against every\ninternet-reachable MOVEit instance Clop could find.
The deeper problem is what MFT software is. Products like MOVEit,\nGoAnywhere, and Cleo exist specifically to sit at the network edge,\naccept authenticated (and sometimes unauthenticated) file uploads from\nexternal partners, and hold the resulting files — often full of PII,\nfinancial records, or healthcare data — until someone downstream picks\nthem up. That's three attractive properties stacked on one box: it has to\nbe internet-facing by design, it processes untrusted external input as its\ncore function, and the data sitting on it is exactly what a data-extortion\ncrew wants. A vulnerability class that would be a moderate finding on an\ninternal app becomes catastrophic on an MFT server because the \"why would\nan attacker target this\" question answers itself.
\nClop's MOVEit campaign wasn't a slow-burn intrusion — CISA and the FBI\ndescribed mass, opportunistic exploitation across a few days before\nswitching to the extortion phase (CISA). Confirmed victims spanned government agencies (multiple U.S. federal\nagencies among them), airlines (British Airways, Aer Lingus), payroll\nprovider Zellis and its downstream customers, the BBC, and the government\nof Nova Scotia — the breadth reflecting how MOVEit sits inside countless\nunrelated organizations' back-office data flows (BankInfoSecurity). By late October 2023, tracking firm Emsisoft put the confirmed count at\n2,559 organizations and over 66 million individuals, with later tallies\nclimbing past 2,700 organizations and roughly 90+ million people as more\ndownstream disclosures rolled in through 2024 (Cloudskope).
\nNotably, Clop didn't encrypt anything. This was pure data-theft-and-extort:\nsteal the files, then list victims on a leak site with a payment deadline.\nThat's become the default MFT playbook — encryption is optional, exposure\nof the stolen files is the leverage, and it's fast to scale because the\nsame exploit chain works against every unpatched instance simultaneously.\nIt repeated almost exactly with the earlier GoAnywhere MFT SQLi/RCE\n(CVE-2023-0669) and again in December 2024 against Cleo's Harmony, VLTrader,\nand LexiCom products, where Clop chained CVE-2024-50623 and\nCVE-2024-55956 — the second flaw shipped because the first patch was\nincomplete — to deploy a Java backdoor and hit organizations that thought\nthey'd already remediated (Rapid7, BleepingComputer).
\nPatch immediately and don't stop there — Progress shipped fixes for\nCVE-2023-34362 within days, but two related SQLi flaws\n(CVE-2023-35036, CVE-2023-35708) surfaced shortly after in the same code\npaths, the same \"one patch wasn't the whole story\" pattern seen later with\nCleo. Treat a single advisory as the start of a remediation window, not\nthe end.
\nFor detection, CISA's advisory (AA23-158A) published concrete IOCs worth\nhunting for regardless of which MFT product you run: unexpected .aspx\nfiles in the web root (human2.aspx for MOVEit specifically), new/unknown\nadmin or service accounts created around the exploitation window, SQL\ninjection patterns and anomalous query volume in application logs, and\noutbound connections to unfamiliar IPs shortly after suspicious file\nactivity (CISA). CISA also published YARA and Sigma rules for LEMURLOOT specifically —\nrun them even after patching, since a compromise that predates your patch\nwon't be undone by it.
Architecturally, stop treating MFT servers like ordinary web apps. Segment\nthem into their own network zone with tightly scoped egress — a file\ntransfer server has no legitimate reason to be initiating arbitrary\noutbound connections, so alerting on unexpected outbound data flows is one\nof the highest-signal detections available. Put a WAF in front of the web\ninterface and use it to block known exploit patterns while you patch.\nMinimize what's actually exposed to the internet: if partners can reach\nyou over a VPN or IP allowlist instead of the open web, do that. And treat\nthe underlying files as sensitive at rest — encrypt them, and don't let\nthe transfer server itself be the only thing standing between \"in transit\"\nand \"exfiltrated.\"
\nMOVEit, GoAnywhere, and Cleo aren't unrelated incidents — they're the same\nshape of failure recurring because MFT software occupies a structural sweet\nspot for attackers: internet-facing by requirement, processing untrusted\ninput by design, and holding exactly the data a data-extortion crew wants\nto steal. That combination doesn't exist for MFT alone — it applies to any\nedge appliance that ingests external data and holds something valuable\nafterward. If you run one, the operative question isn't \"has this been\npatched recently,\" it's \"what happens to every file that's touched this\nbox if it's compromised tomorrow\" — and whether your network segmentation,\negress monitoring, and patch cadence would actually catch it before it\nbecomes the next mass-breach headline.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "security", "cve", "ransomware", "appsec" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-public-s3-bucket-exposure/", "url": "https://jstgtech.com/blog/2026-08-10-public-s3-bucket-exposure/", "title": "The 198M-voter S3 leak that still explains most breaches", "summary": "A 2017 misconfigured S3 bucket exposed 198 million voter records, and the same public-bucket misconfiguration still causes new breaches every year.", "content_html": "Most S3 exposure incidents don't require a zero-day, a phishing email, or\neven much skill — just someone who knows how to type an S3 URL into a\nbrowser. In June 2017, UpGuard security researcher Chris Vickery found\nexactly that: a bucket named dra-dw — \"Deep Root Analytics Data\nWarehouse\" — sitting fully public on the open internet, no password and no\nauthentication anywhere in front of it. Inside was 1.1 terabytes of data on\nnearly every registered voter in the United States: 198 million records\nwith names, dates of birth, home addresses, phone numbers, and modeled\nethnicity and religion scores, compiled by contractors working for the\nRepublican National Committee (Gizmodo, UpGuard). Deep Root Analytics had, in\nUpGuard's account of the incident, simply set the bucket's permissions to\npublic instead of private (CyberScoop). That's a 2017 incident — old\nenough that AWS has since shipped several generations of guardrails\nspecifically built to prevent it — and yet the identical failure mode, a\nbucket flipped to public and found by a researcher scanning the internet\nrather than by the company that owns it, keeps showing up in breach\nreports nearly every year since. That's the real story here: not one\nincident, but a misconfiguration class that refuses to die.
S3 buckets are private by default. Getting one exposed takes a deliberate\n(if often thoughtless) action, and over a decade of these incidents the\npaths to that action have stayed remarkably consistent. Legacy ACLs are\nthe classic one: S3's original access-control-list model lets you grant\nread or write access to predefined groups, and two of those groups are\nroutinely confused for \"my organization\" when they actually mean \"the\nentire internet\" or \"any authenticated AWS account on the planet\" — not\nyour customers, not your team, literally anyone with an AWS account.\nBucket policies cause the same damage a different way: a policy meant to\nscope access to a specific role or CloudFront distribution gets written\nwith a wildcard principal (\"Principal\": \"*\") or an overly broad condition\nthat accidentally satisfies for anonymous requests too. And a large share\nof real-world exposures trace back to third-party tooling — backup\nutilities, data pipelines, static site generators, BI exports — that\ndefaults to a public bucket because it's the path of least resistance for\nthe vendor's quickstart guide, and nobody revisits the setting once it\nworks. None of these require malice. They require a permission model with\nmore than one way to say \"public\" and no default nudge toward noticing\nyou've done it.
The Deep Root leak wasn't found by Deep Root. It was found by a\nthird-party researcher running broad scans for open cloud storage, who\nstumbled onto a predictable Amazon subdomain,\ndra-dw.s3.amazonaws.com (UpGuard). That's the pattern across\nnearly every S3 exposure story of the last decade — Verizon's exposed\ncustomer call records via a third-party vendor, Dow Jones's 2.2 million\nsubscriber records, Accenture's exposed API keys and credentials — the\ncompany almost never discovers its own leak. It's found by a security\nresearcher, a journalist, or occasionally a criminal, all of whom have the\nsame tooling: internet-wide bucket-name scanners that turn \"is this bucket\npublic\" into a solved, automatable question. Once a bucket is enumerable,\nthe blast radius is everything in it, immediately, with no exploit chain\nrequired — the exposure is the vulnerability. For Deep Root that meant PII\non 198 million Americans sitting downloadable for at least several days\nbefore Vickery's report reached the company and the bucket was locked down\n(CyberScoop).
AWS's answer to this exact failure mode is S3 Block Public Access,\nlaunched in November 2018 specifically in response to years of incidents\nlike this one — it's a set of four switches, settable at the bucket or the\nwhole-account level, that override any ACL or bucket policy trying to\ngrant public access, so a misconfigured policy simply can't take effect\n(AWS). As of April 2023, AWS made this the default for every new bucket and\ndisabled ACLs by default account-wide, which closes off the legacy-ACL\npath that caused Deep Root's leak unless someone deliberately opts back\nin. Layer on top of that: IAM Access Analyzer for S3, which\ncontinuously flags buckets reachable from outside your account or\norganization, including via cross-account bucket policies that Block\nPublic Access alone won't catch; an AWS Config rule\n(s3-bucket-public-read-prohibited / s3-bucket-public-write-prohibited)\nthat continuously evaluates every bucket against the policy you've set,\nrather than only at creation time; and an SCP at the AWS Organizations\nlevel denying any principal from disabling Block Public Access or\nattaching a public bucket policy in the first place, so an individual\naccount or a rushed engineer can't quietly opt back into exposure.\nFinally, turn on CloudTrail S3 data events (off by default, since\nthey're high-volume) for buckets holding anything sensitive — object-level\nGetObject logging is what tells you whether an exposure window was ever\nactually accessed, versus merely theoretically reachable, which matters\nenormously for incident scoping and breach-notification decisions.
This misconfiguration has been headline news since at least 2017 —\nVerizon, Dow Jones, Accenture, WWE, Pentagon contractor INSCOM, and Deep\nRoot Analytics all made the same mistake within about a twelve-month\nstretch — and it's still a recurring line item in breach reports today,\nyears after AWS built free, one-click guardrails specifically to stop it.\nThe reason isn't that the fix is hard; it's that S3's original design let\n\"public\" be an opt-in property of individual objects and policies, which\nmeans it only takes one wrong click, one stale ACL, or one default-public\nthird-party tool to undo. Block Public Access flips that assumption: it\nmakes private the property you have to fight to escape, at the account\nlevel, instead of the property you have to remember to defend at the\nobject level. Every org still running without Block Public Access\nenforced by SCP, without Access Analyzer alerting, and without Config\nrules gating drift is one misclick away from being next month's version of\nthis story.
\n", "date_published": "2026-08-10T00:00:00.000Z", "tags": [ "security", "aws", "s3", "cloud" ] }, { "id": "https://jstgtech.com/blog/2026-08-10-runc-containerd-escape/", "url": "https://jstgtech.com/blog/2026-08-10-runc-containerd-escape/", "title": "CVE-2024-21626 and runc's shared-kernel blast radius", "summary": "A leaked file descriptor in runc, and a fresh 2025 trio of procfs races, show what \"container isolation\" really guarantees — and what it doesn't.", "content_html": "Every container on a multi-tenant host is a guest sharing one kernel with\nevery other tenant. Namespaces and cgroups make that kernel look partitioned,\nbut they don't partition the kernel's code — a bug reachable from inside a\ncontainer is reachable from the same privilege level the kernel runtime\nitself trusts. CVE-2024-21626, the \"Leaky Vessels\" flaw in runc, is the\nclearest recent proof: a leaked file descriptor let a malicious image escape\nits container and land as root on the host, no exploit chain required beyond\na crafted Dockerfile (Snyk, The Hacker News). And it wasn't a one-off — a fresh batch of runc container-escape CVEs\nlanded in late 2025, hitting the exact same trust boundary from a different\nangle.
runc is the low-level OCI runtime underneath Docker, containerd, and most of\nKubernetes' container execution path — it's the thing that actually calls\npivot_root, sets up namespaces, and execs the container process. CVE-2024-21626\ncame from an order-of-operations bug: runc could leak an internal file\ndescriptor referencing the host's working directory before it finished\nwalling the process off with pivot_root. A malicious image that set its\nWORKDIR to something like /proc/self/fd/7 could ride that leaked fd\nstraight into a directory on the host filesystem, escaping the container\nrootfs entirely (GitHub PoC, Red Hat).
The 2025 trio — CVE-2025-31133, CVE-2025-52565, and CVE-2025-52881,\ndisclosed by a SUSE researcher in November — is the same story with a\ndifferent mechanism: race conditions in how runc mounts /dev and enforces\n\"masked paths\" under /proc. An attacker who controls container startup\nconfig can swap /dev/null or /dev/pts/$n for a symlink pointing at a\nsensitive procfs file milliseconds before runc bind-mounts it, tricking the\nruntime into mounting host /proc paths read-write inside the container, or\nredirecting writes meant for a scoped procfs entry to arbitrary host paths\nlike /proc/sysrq-trigger (Sysdig, OCI advisory). Different bug class, same underlying truth: the isolation boundary is\nenforced entirely by kernel code that every container process can call into,\nand a single logic slip anywhere in that code path breaks the boundary for\neveryone sharing the kernel.
An attacker who lands one of these isn't stealing data from a container —\nthey're getting arbitrary code execution on the host, at whatever privilege\nthe container runtime holds, which is typically root. On a shared build farm,\nCI runner, or multi-tenant Kubernetes node, that one escape gives an\nattacker every other container's filesystem, secrets mounted into other\npods, the kubelet's credentials, and a jumping-off point to the rest of the\ncluster. Leaky Vessels was especially nasty for CI/CD because it triggers\njust by building a malicious image — you don't need to run untrusted code\nin production, you just need your pipeline to pull an attacker-controlled\nbase image or Dockerfile and build it (Palo Alto Networks).
\nWorking proof-of-concept exploit code for CVE-2024-21626 has been public on\nGitHub since shortly after disclosure, which is exactly the scenario that\nturns a CVSS score into an operational problem — no 0-day skill required,\njust an unpatched runtime and a way to get a container built or started. The\n2025 trio requires a bit more precision (winning a mount race), but the\noutcome is the same: root on the node, and from there, lateral movement\nacross whatever else that node was trusted to isolate.
\nPatch the runtime, not just the orchestrator sitting on top of it — Docker\nand Kubernetes ship runc bundled, so \"update Kubernetes\" doesn't automatically\nmean \"update runc\" if you're on a vendored or older build:
\nBeyond patching, this bug class is exactly why \"container\" and \"security\nboundary\" shouldn't be treated as synonyms for anything you don't trust.\nDefense in depth that actually helps here: run rootless containers so a\nkernel-level escape lands as an unprivileged host user instead of root;\napply seccomp and drop capabilities aggressively so even a successful\nescape has less to work with; and for genuinely untrusted workloads — public\nCI runners, multi-tenant SaaS sandboxes, anything running code you didn't\nwrite — put a real boundary under the shared kernel with gVisor (a\nuserspace syscall shim the exploit has to get through first) or Kata\nContainers (a real VM boundary per workload, so a kernel exploit only pops\nthat one micro-VM).
\nrunc, containerd, and every OCI-compliant runtime built on the shared-kernel\nmodel give you process isolation, not a security boundary against a\nsufficiently novel kernel bug — and this class of vulnerability proves that\ngap isn't hypothetical, it's recurred across at least two unrelated bug\nclasses in two years. That's fine for internal services running your own\ntrusted images. It's a real risk for anything that runs code you don't\ncontrol: public CI, contributor-submitted builds, multi-tenant platforms.\nFor those, the honest question isn't \"did we patch runc\" — it's \"why are we\nrelying on a shared kernel to isolate an adversary in the first place,\" and\nwhether a VM-backed runtime like Kata or a syscall-filtering sandbox like\ngVisor should be the default instead of the exception.
If you run a self-hosted CI/CD server, stop reading and go patch it. That's the headline today — the rest is a mix of AWS platform news and more actively-exploited flaws in tools a lot of us have sitting in our stacks.
\nCISA added CVE-2026-63077, a critical (CVSS 9.8) unauthenticated remote code execution flaw in JetBrains TeamCity On-Premises, to the Known Exploited Vulnerabilities catalog on August 5 (CISA, The Hacker News). An attacker can abuse the agent polling protocol to skip authentication entirely and run arbitrary OS commands as the TeamCity server process — which means your build agents, artifacts, and any credentials TeamCity holds for deploying into AWS are all in play. JetBrains has patches out (upgrade to 2025.11.7 or 2026.1.3, per JetBrains' advisory); federal agencies were given until August 8 to remediate under BOD 26-04. If your TeamCity server has any path to the public internet, treat this like the SolarWinds-style CI/CD compromise scenario it is and patch today, then audit what credentials that server has been handing out.
\nThe same CISA update cycle also added CVE-2026-9198, a code injection bug in Langflow (the visual builder a lot of teams use to prototype LLM pipelines) that gives unauthenticated RCE on default installs — telemetry shows 650+ exploitation attempts from 244 unique IPs since early July (The Hacker News). And CVE-2026-34486, a missing-encryption flaw in Apache Tomcat's EncryptInterceptor, is reportedly being hit by an AI-orchestrated, China-nexus automated attack campaign. If you've stood up Langflow for an internal AI experiment and forgotten about it, that's exactly the kind of exposed dev tool this campaign is scanning for — take: audit anything you spun up \"just to try AI stuff\" six months ago and never locked down.
Less urgent, more useful: AWS Lambda functions outside a VPC with 2 GB+ of memory now scale network bandwidth proportionally, from 625 Mbps at 2 GB up to 3,000 Mbps at 10 GB, at no extra charge (AWS). That's a meaningful jump for anything doing large payload transfers, S3 streaming, or bursty data-heavy work in Lambda — previously you were capped at 625 Mbps regardless of memory. Catch: it's not automatic. You have to request the \"Network bandwidth per execution environment\" quota bump via Service Quotas before your functions see it, so if you have latency-sensitive Lambdas moving a lot of data, it's worth requesting now even if you don't need it yet — quota changes aren't instant.
\nCloudWatch now offers managed Prometheus collectors that auto-provision and scale to pull OpenTelemetry/Prometheus metrics from EKS, EC2, ECS, MSK, and OpenSearch without you running your own collector fleet — worth a look if you're maintaining a hand-rolled Prometheus scraping setup on EKS. And separately, N-able confirmed active exploitation of an N-central authentication bypass (CVE-2026-18577) that let an attacker pivot from a compromised N-central server into managed endpoints — a reminder that RMM/MSP tooling is a high-value target precisely because of the blast radius one compromised instance gives an attacker.
\nToday's theme is \"your build and automation tooling is the target, not just your app.\" Patch TeamCity if you run it, sweep for forgotten Langflow instances, and grab the free Lambda bandwidth quota bump while it's on your mind.
\n", "date_published": "2026-08-09T00:00:00.000Z", "tags": [ "roundup", "aws", "security" ] } ] }