After taking my time playing Automation Empire and really loving it, I got to a point where the tax system really destroyed it for me. The way they designed it was that tax was based on 2% of lifetime earnings for the save game that you’re playing. They claim that this made players have to work consistently to expand their profitability and push to finish maps, but the general sentiment in the forums (and for me) is that it ruins the fun of taking your time and exploring what’s possible in the game. They claim there’s a reset in the algorithm, but it doesn’t actually work.
[Read More]Google's SRE Book on Eliminating Toil
For everything that Google does which I have strong opinions about, some of their SRE training and information is pure gold. I just finished reading the chapter on eliminating toil and it really hits home for me.
If a human operator needs to touch your system during normal operations, you have a bug. The definition of normal changes as your systems grow.
- Carla Geisser, Google SRE
So, the more time you spend keeping things running, the less time you have to make them better - for you and your end users.
[Read More]Troubleshooting the O365 Message Reporting Add on for Splunk
Periodically this thing dies on me. It happened again, so here’s my notes.
Messages stopped coming in, I got an alert, and found this log:
2019-11-11 13:53:56,750 DEBUG pid=20951 tid=MainThread file=connectionpool.py:_new_conn:809 | Starting new HTTPS connection (1): reports.office365.com
2019-11-11 13:53:57,019 DEBUG pid=20951 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-10-22T02:53:08.114678Z'%20and%20EndDate%20eq%20datetime'2019-10-22T03:08:08.114678Z' HTTP/1.1" 200 216
2019-11-11 13:53:57,022 DEBUG pid=20951 tid=MainThread file=base_modinput.py:log_debug:286 | No messages returned. Setting max date to 2019-10-22 02:54:08.114678
The “No messages returned.” bit was the kicker. Lies!
[Read More]Nobody Dies in Longyearbyen
A fascinating look at the northernmost city in the world, touched by climate change in ways you wouldn’t expect.
[Read More]Facebook Engineering and Ssh Keys
This post on the facebook Engineering blog about scalable and secure access with SSH really makes me wonder how this’d be doable at-scale, without a fleet of developers to build your own system to do it.
The advice at the end is probably the most important information any AAA system team can take heed of:
A few parting words of advice: When you build your CA, be it a small script or a complex system, make sure you keep track of all certificates you issue. If you find yourself in the unfortunate situation of having a compromised certificate (and its respective private keys) and you don’t know how to revoke them, your last resort is to rotate the entire CA. If you end up having a programmatic CA, consider having short-lived certificates, e.g., 24 hours. This shortens the window of opportunity for an attack if you experience a compromise.
[Read More]
DictWriter instance has no attribute '__exit__'
So I’m messing around with a DictWriter and was trying to use the typical “with” syntax I use with short-lived things in python… and got this error:
Error: DictWriter instance has no attribute '__exit__'
Turns out, that’s not really a thing, and I should have followed the example. Don’t judge me please for using Py2, Splunk hasn’t caught up (yet).
[Read More]git xcrun error after catalina upgrade
It’s been pretty smooth after upgrading to catalina, but I got this error when trying to run git this morning…
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
Easily fixed…
xcode-select --install
Let it do the thing and you’ll be back in business.
Updates:
- If you’re still getting the “bash is going away” prompt, add
export BASH_SILENCE_DEPRECATION_WARNING=1to~/.bash_profile) - terraform wouldn’t load because it’s not signed by something Apple likes, so here’s an article on how to fix terraform - tl;dr find the binary in the finder and right-click-open-allow it.
purplecon 2019
The 87th annual purplecon was delightful and fun. Eschewing the typical black hoodies and replacing them with sparkles was an amazing choice and drove the friendly, welcoming feel of the entire con.
All talks were required to be:
- positive,
- defensive, and
- actionable.
Which means they’re not just stunt hacking or dropping 0-days - they’re designed to improve the state of the art. The “great archive” is going to be a text archive summary of the talks, so…
[Read More]sssd and the sudo alerts
So… I kept getting a message like this every. single. time. I. used. sudo.
Subject: *** SECURITY information for server.domain.example.com ***
Message: server.domain.example.com : Oct 19 14:17:50 : yaleman : problem with defaults entries ; TTY=pts/0 ; PWD=/home/yaleman ; USER=root ;
Doing some searching, it turns out it’s some issue with local accounts and an interaction between sudo and sssd where defaults aren’t being presented to sssd from FreeIPA/LDAP. It’s more a warning than a bug, but it’s freaking annoying.
Cleaning Up CrowdStrike Linux Files
Sometimes on Linux hosts, CrowdStrike has a bit of a weird issue with filling up /opt/CrowdStrike with old versions. The fix they gave us was… a bit special, so I came up with my own.
sudo find /opt/CrowdStrike -type f -name "KernelModuleArchive*" -not -wholename "$(readlink -f /opt/CrowdStrike/KernelModuleArchive)" -exec rm "{}" \;
sudo find /opt/CrowdStrike -type f -name "falcond*" -not -wholename "$(readlink -f /opt/CrowdStrike/falcond)" -exec rm "{}" \;
sudo find /opt/CrowdStrike -type f -name "falconctl*" -not -wholename "$(readlink -f /opt/CrowdStrike/falconctl)" -exec rm "{}" \;
sudo find /opt/CrowdStrike -type f -name "falcon-sensor*" -not -wholename "$(readlink -f /opt/CrowdStrike/falcon-sensor)" -exec rm "{}" \;
For each of the different file types it makes, there’s a “versioned” file, such as falcon-sensor7303 which is symlinked as /opt/CrowdStrike/falcon-sensor. When the “real” files update to new versions, they don’t clean themselves up. The above four lines looks for the “bad” files and filters out the existing “current” file using readlink to identify the canonical destination of the symlink.