lever changed</th>	chat</th>	long doc</th>	code</th></tr></thead>
published config</strong> (autotune drafter)</td>	12.3</strong></td>	6.8</strong></td>	15.6</strong></td></tr>
drafter, fixed draft 1</td>	15.6 (+27%)</td>	9.8 (+44%)</td>	15.9 (+2%)</td></tr>
drafter, fixed draft 2</td>	16.1 (+31%)</td>	7.4 (+8%)</td>	17.8 (+14%)</td></tr>
drafter, fixed draft 3, autotune off</td>	13.9 (+13%)</td>	7.1 (+4%)</td>	18.3 (+17%)</td></tr>
drafter off entirely</td>	12.2 (wash)</td>	10.5 (+54%)</td>	12.2 (−22%)</td></tr>
flash attention off</td>	6.7 (−46%)</td>	4.3 (−37%)</td>	7.5 (−52%)</td></tr>
threads `-t 4</code></td>`	7.9 (−35%)</td>	5.0 (−28%)</td>	9.5 (−39%)</td></tr>
threads `-t 16</code></td>`	10.8 (−12%)</td>	7.1 (+3%)</td>	16.2 (+4%)</td></tr>
run-time-repack off</td>	11.8 (−4%)</td>	9.2 (+35%)</td>	13.7 (−12%)</td></tr> </tbody></table> Everything else I tried, --mla-use</code>, --cpu-moe</code>, --merge-up-gate-experts</code>, --no-kv-offload</code>, the -sm graph</code> cluster, lands within a few percent of the published config on chat and code, which is to say inside the noise. The long-document column is messier than the other two and I will come back to why in the repack section. Read the negatives as the interesting part. Most of these levers do nothing or cost you, and the work is concentrated in flash attention, the physical-core thread count, and the drafter once it is configured the right way.</p> The drafter is workload-dependent, and one setting was wrong</h2> Speculative decoding is the most tuned part of the config, and it is where I found the most.</p> The first finding is that the drafter is not a free win across the board. Measured against turning it off entirely:</p> On chat it is roughly a wash at the default setting.</li> On code it is worth about twenty-eight percent</strong>. A real win.</li> On the long document it makes things slower. Turning the drafter off is fifty-four percent faster. Summarization is hard to predict, the drafter’s acceptance rate sits down around 0.15. A draft that gets rejected most of the time is mostly overhead.</li> </ul> So the drafter wants to be gated by workload. On for code, off for long-context work. The config has it on globally, which quietly costs you about a third of your throughput every time you summarize something. This was not something I had clocked when I wrote the original post, and it is the single most useful thing in this one.</p> The same flag being a win, a wash, and a loss depending on the prompt is worth sitting with, because it is a tiny hand-measured example of a routing gap. The model already routes internally: a mixture-of-experts gate sends every token to the experts it judges most useful, learned and conditional, per token. But that gate optimizes for the next token, not for which inference machinery pays off on a given kind of work. The choice I am making by hand here, speculate on code, do not speculate on the summary, is a second and coarser routing decision sitting one level up, per request instead of per token, and the config hardcodes one answer to it and applies that everywhere. You could imagine a cheap classifier in front of the engine that picks the decoding policy the way the expert gate picks experts, and --spec-autotune</code> is a clumsy first stab at that idea, aimed at a single knob and, on this workload, aimed badly. I am not saying the expert routing is broken. I am saying there is a routing decision above it that my numbers say is worth making, and I made this one with a shell script and a stopwatch… metaphorically. The stopwatch is a metaphor to be clear.</p> The second finding is a setting I had wrong. In the Xeon post I described --spec-autotune</code> as the way to squeeze the most speed out of speculation. Over these generations it is the worst speculation setting I tested. Every fixed --draft-max</code> beats it.</p> chat: autotune 12.3 tokens per second, a fixed draft of 2 gets 16.1. Twenty-five flags, and the biggest single win in the file was deleting one of them and typing a number.</li> code: autotune 15.6, a fixed draft of 3 with autotune off gets 18.3.</li> long document: autotune 6.9, a fixed draft of 1 gets 9.8.</li> </ul> The mechanism is acceptance rate. A shorter, fixed draft gets accepted by the verifier more often. A draft of 1 on chat sits at 0.74 acceptance against autotune’s 0.37, and acceptance is the entire currency speculation trades in. The autotuner spends the generation hunting for a draft length, and over a couple of hundred tokens it never settles, so the average gets dragged below the steady state.</p> The honest caveat, because it matters: 256 tokens is a short generation. It is entirely possible that over a long session, a couple of thousand tokens, the autotuner converges up to the fixed rate and my numbers are simply under-reporting it. I have a test designed to settle that and I have not run it yet. So the careful claim is narrow: at short generation lengths, on this workload, a fixed draft beats autotune. Whether that survives at length is the next experiment. Either way it is a free win today, because picking a number puts the interactive prompts up around sixteen to eighteen tokens per second.</p> Flash attention is the one doing the heavy lifting</h2> If you keep one flag from the whole spell, keep this one. Flash attention is worth roughly two times. Turn it off and chat falls from 12.3 tokens per second to 6.7, code from 15.6 to 7.5, the long document from 6.8 to 4.3. Forty-six, fifty-two, and thirty-seven percent gone, from a single flag.</p> It also turns out to be holding up the drafter, which I did not expect. With flash attention off, speculative acceptance collapses to near zero. So the flag doing the most obvious work is also quietly propping up the flag I just spent a whole section on. That is the kind of dependency you only see when you turn things off one at a time.</p> Threads want physical cores, and there is a wall behind them</h2> This one I got right in the original, and now I have the number to back it. -t 8</code>, one thread per physical core, is the sweet spot, holding chat at 12.3 tokens per second. -t 16</code>, using all sixteen hyperthreads, slips to 10.8, slightly worse, because the work is waiting on memory and extra threads add scheduling overhead without adding bandwidth. The cores are stalled on DDR3, not on each other. -t 4</code> is about thirty-five percent slower on decode and closer to forty-five percent slower on prefill, which is the memory wall poking through.</p> While we are here, that wall is worth making concrete, because it tells you when to stop fiddling with flags. Gemma 4 26B-A4B activates about four billion parameters per token, which at Q8_0 is roughly 4.25 gigabytes read for every token generated. Divide that into the memory bandwidth this box can sustain and you get a ceiling around ten to eleven tokens per second for the verifier on its own. That is almost exactly where the thread sweep flattens out, which is how you know it is a real physical wall and not a tuning problem. Past that point, no flag saves you. The only move left is to read fewer bytes per token, which is the entire subject of the next post.</p> Repacking earns its keep on prefill</h2> --run-time-repack</code> mostly helps prefill, about nineteen percent on the chat and code prompts, with a small and noisy effect on decode for those two. It spends a few seconds at startup physically rearranging the weight matrices to match how the CPU wants to read them. On a Q8_0 model that is a modest, real win, and worth the startup cost.</p> There is one honest oddity the board above will have flagged. On the long document, turning repack off</em> reads about thirty-five percent faster on decode, which is the opposite of what a prefill optimization should do. It is not a prefill effect. It is the drafter’s long-document acceptance jumping, from around 0.15 to around 0.41, when the weights are not repacked. I do not yet know why repacking the tensors should move acceptance at all, so I am flagging it as a loose thread rather than dressing it up as a result. It is also most of why the long-document column on the board is noisier than the other two.</p> It also has a much more interesting and much worse behavior on a different quant, which is a cliffhanger for the next post.</p> The flags that do not take for free</h2> This is the half I warned you about. None of it is dramatic, and most of it is not “useless flag,” it is “flag that needs something you might not have”. I ran each one as its own measurement specifically to pin down which is which, with the engine log as evidence rather than a hunch.</p> --mlock</code> needs a persistent memory-lock limit, and it worked last time.</strong> I have to be careful with this one, because it is easy to tell the wrong story about it. mlock pins the weights in RAM so the kernel cannot page twenty-five gigabytes of model out to disk and drop your speed to zero. In the Xeon post it worked, because I had raised the memlock ulimit on that box. For this round of measurements the box had been rebooted, the ulimit had quietly reverted to its tiny default, and mlock fell back to running unpinned behind a one-line warning. That is not the flag doing nothing. That is my host setup not surviving a reboot. The fix is to raise the limit persistently</em>, in the systemd unit or limits.conf</code>, not in a shell that dies when you log out. On a box with no swap, like this one, pinning is belt-and-suspenders anyway, but on anything with a swap device it is the difference between steady output and a sudden cliff.</p> -sm graph</code> and its friends want more than one device.</strong> The -sm graph -smgs -sas -mea --split-mode-f32</code> cluster is about splitting the work across multiple processors or sockets. This box is one CPU with no GPU, so there is nothing to split across, the engine says exactly that, and it falls back to a sequential layer split. I keep these in as a forward bet, because the moment you put this on a two-socket board they start mattering, and -sas</code> is there for precisely that. On one socket they are inert, and that is expected, not broken.</p> --cpu-moe</code> and --merge-up-gate-experts</code> fire but do not move the needle here.</strong> These two do engage. The log confirms the fused expert path is active. But switching them off lands inside the run-to-run noise on this hardware and workload. So the honest verdict is narrower than the rest: they take, they just do not measurably change decode speed on this box. They may well earn their place on a different CPU. Engaged and useful are not the same property, and the only way to tell them apart is to measure.</p> --mla-use</code> is the one that is genuinely not wired.</strong> Here is the second thing I had wrong in the original. I presented --mla-use 3</code> as active. It is not. MLA is a DeepSeek architecture feature, it is not implemented for Gemma 4, and the flag is accepted and silently ignored with no log line at all. I swept it from 0 to 3 to be sure. No difference. This is the one in the section you can actually just delete.</p> A general warning against reading any of this too hard. “No measurable difference on my box, on my workload, at these context lengths” is not “useless everywhere”. Several of these earn their keep on other hardware. The claim is deliberately narrow, and the only reason I can sort them at all is that I checked each one against the log instead of trusting the command.</p> The pitfalls</h2> Not everything worth knowing is a tokens-per-second number. Three of these cost me real time, and they are the kind of thing you only hit by running the engine into the ground on purpose.</p> mlock fails silently.</strong> I covered the cause above, but the failure mode earns its own line, because it is nasty. mlock does not error out. It prints one warning and runs unpinned. A config that pinned your model perfectly last week can quietly stop pinning after a reboot, your speed gets unpredictable, and nothing in the output points at the cause. Set the limit in the unit file, and check the startup log actually says it took. Every boot.</p> -sm graph</code> is a landmine that another flag is defusing.</strong> With the drafter on, the engine sees that graph split is unsupported under MTP and quietly downgrades it to a layer split. That downgrade is the only reason the flag is survivable at all. Turn the drafter off, run the same config, and the model refuses to load:</p> error loading model: It is not possible to use split mode 'graph' </span>with less than 2 devices </span></code></pre> So a flag that does nothing useful on this box is also fatal on its own, and it only avoids exploding because a neighbor steps in front of it. You find that out by turning the neighbor off.</p> The slot-zero deadlock that ate an overnight run.</strong> The first full benchmark looked nondeterministic in a way that smelled like a memory death spiral. The chat prompt always finished, but the long document and the code prompt usually hit the timeout, including a 106-character code prompt that should take fifteen seconds. It was not memory. Under --parallel 8</code> the server has eight slots, but the drafter only works on slot zero on this build. Any request that lands on another slot deadlocks the instant it clears the cache, generates nothing, and never lets go:</p> kv cache rm [p0, end) ... id_slot=1 id_task=82 p0=0 <- then nothing, ever </span></code></pre> The server assigns slots by prompt similarity, so which prompt drew the dead slot changed from run to run, which is where the fake randomness came from. Aggregated up, it looked like a resource problem. The per-request log showed a plain control-flow deadlock. Read the log, do not theorize about RAM. The fix is to pin every request to slot zero, which is also exactly what a single interactive user does anyway.</p> Setting a parameter to its own default breaks it.</strong> My favorite. The config passes no -c</code>, so it runs at the model default context of 262144. Under llama-server</code> that default path fails to initialize the drafter and wedges the server: it stays up, holds the port, and never reports healthy, so a naive wait-for-health loop hangs forever:</p> llama_init_from_model: n_batch and n_ubatch cannot both be zero </span>MTP context creation failed </span></code></pre> Pass -c</code> explicitly at any size, including 262144</code>, the exact value the default would have resolved to, and it works. The bug is in the path that handles the missing flag, not in the number. I nearly shipped a wrong explanation for this, a theory about per-slot sizes built on two data points, and only dodged it because the benchmark actually ran the 262144 case instead of assuming it would fail. Run the experiment you think you can predict.</p> So which flags actually mattered</h2> After all of it, the honest shape of the config.</p> The flags doing real, measurable work on this box are flash attention by a mile, the physical-core thread count, run-time repack for prefill, and the drafter for code with a fixed draft length. mlock belongs in that list too, as long as you set the limit so it survives a reboot. The rest is either waiting on hardware I do not have, firing without moving the needle, genuinely not wired, or a landmine held in check by a neighbor. Decode runs from about seven tokens per second on the long document up to sixteen on code, which is reading speed, and gating the drafter and fixing the draft length lift both ends from there.</p> I knew the rough shape of this before I started. I said half the flags would not take. What I did not have, and now do, is the sorted version: which half, the two settings I had wrong, the few that carry the whole thing, and the exact cost of each. Twelve of the twenty-five come out entirely, and the honest part is that deleting them changed nothing, because they were deadweight, and removing deadweight is free. The few tokens a second I picked up did not come from the cull. They came from fixing two flags I kept: trading the autotuner for a fixed draft length, which on its own takes chat from 12.3 to 16.1, and gating the drafter by workload. The config I keep is a good deal shorter than the one that went up on Hacker News, but the shortening and the speedup are two different stories, and only one of them is about deletion.</p> How to check your own</h2> The transferable part of this is not my flag list, which is specific to this box and will go stale the next time the fork moves. It is the procedure, and there are four steps.</p> Check that the flag fired. For everything you are relying on, find the line in the engine log that proves it engaged: the init message, the = 1</code>, the loaded-drafter block. There are three outcomes, not two. It engaged, it was declined with a warning, or it was silently ignored with no line at all. That last one, like --mla-use</code> here, is the easiest thing in the world to cargo-cult, because it costs nothing, does nothing, and looks exactly like success.</p> Turn off one thing at a time. The whole approach only works if you change a single flag per run, off a config that already works. The marginal contribution given everything else is the number that is actually true. Change five flags at once and you learn nothing.</p> Gate by workload. The same drafter was a win for code and a loss for summarization. If your config hardcodes a choice that depends on what you are generating, you are leaving throughput on the floor for half of what you do.</p> Know your ceiling. Active parameters times bytes per weight divided by memory bandwidth gives you the verifier’s solo speed limit. On this box that is around ten to eleven at Q8_0. Once you are there, you are not adding flags anymore, you are reading bytes, and the only way down is fewer of them.</p> What is next</h2> Every flag in this whole exercise either does nothing about the memory wall or chips at it sideways. The one lever that hits it head on is the one I never touched. Everything here ran at Q8_0.</p> Dropping the verifier to a smaller quant reads fewer bytes per token directly, which is the most honest way to buy speed on a memory-bound box, and it should be a clean win. It mostly is. But it also handed me the best benchmark trap I have ever walked into, where the single fastest config I measured was emitting pure garbage, and was fast precisely because it was garbage. That is the next post.</p> And yes. The next post is where we confirm my suspicion that this can run past 20 TPS.</p> Thanks for reading. If you want, you can stop reading here. I’ll add some meta commentary to the end, so those that don’t care can skip it. To those, see you all next time! o/</p> Meta Comments on this series</h2> I didn’t plan to get lost in these weeds… but here we are.</p> Since last time, I’ve made sure to address the comments about link colors to make this WCAG compliant. Another big outstanding comment was that there was a clear lack of numbers.</p> I’ve tried to benchmark things here, but as it turns out, the combinatorial explosion of benchmarking everything is beyond my current compute and time constraints.</p> I’m glad about all the positive feedback I’m getting about this, and I keep ending up writing more. My plan was to “just” write a follow up with numbers, but then that turned into it’s own rabbit hole. And just before I shipped it, I thought “what if I looked into the difference between difference quantizations”… and of course, that gave enough content for an entirely separate post (that I think will likely be the one with the strongest numbers so far, I can’t wait to show it once I’ve done the final polish).</p> It’s important to remember that I am NOT an ML engineer. I’m here in the weeds with you, dear reader, and I may be saying things that aren’t correct. So I really encourage anyone with the interest to actually try and replicate some of this.</p> Point is. If something is wrong, I can probalby fix it. If you think I’m saying something stupid, tell me, that would be a great post. And by all means, apply this. Till next time.</p> A 10 year old Xeon is all you need Christina Sørensen — Mon, 01 Jun 2026 00:00:00 +0000 The previous post</a> covered getting Gemma 4’s MTP drafters quantized and paired with a verifier. This one is about running the result on a machine that has no business running it.</p> I have a recycled server. To its credit, it has a whopping 128 GB RAM, but it’s DDR3… That RAM is 5-6 times slower than the current best laptop ram. It also has a single Intel Xeon E5-2620 v4 from 2016, which is about 5 times slower than my laptops CPU…</p> Oh, and as I did mention, we have no GPU</strong>. And no, the Xeon does not have an integrated GPU.</p> But, just hear me out…</p> If we were to just break out ollama here, well… as explained in earlier blog posts, we can’t. And we’d be lucky if we could in 6 months when they add support for the model we need, if they ever do. Might be they never do. And even still, ollama simply doesn’t expose enough knobs for us to ever make this run well, neither does even the standard llama-cpp</code>.</p> But. Why would that stop us?</p> I’ve recieved feedback that some of the previous posts were too high level, I’ll try to make things as clear as reasonably possible here. If you’re a tech worker, or a Linux enthusiast that has built a computer and used something like ChatGPT, most of this should be approachable.</p> </blockquote> So, just to really set the stage fully. The hardware, per lscpu</code>:</p> CPU:</strong> Intel Xeon E5-2620 v4 @ 2.10 GHz</li> Cores:</strong> 8 physical, 16 threads</li> Instruction sets:</strong> AVX2 (no AVX-512, no AVX-VNNI, no BF16)</li> Cache:</strong> 20 MiB L3, 2 MiB L2 total</li> Memory:</strong> 128 GB DDR3</li> GPU:</strong> none</li> </ul> For LLM inference, memory bandwidth is the limiting resource. Every token generated requires hauling gigabytes of weights from RAM into the CPU cache.</p> When you use a tool like ChatGPT and watch the text stream onto your screen word by word, you are watching the “decoder pass”. During this phase, the model generates the output one piece (or “token”) at a time.</p> In this step, the system’s raw processing power is rarely the bottleneck. Instead, the limitation is memory bandwidth. To calculate that next word, the processor has to constantly pull massive amounts of data. That data is the “weights” that contain the model’s learned knowledge. It moves this from memory into the compute cores.</p> The processor executes the required matrix calculations so quickly that it is left sitting idle, waiting for the hardware to physically move the next chunk of weights across the memory bus. In traditional software terms, decoding is heavily memory-bound, not compute-bound.</p> This is the so called “memory wall”, one of the single biggest performance hurdles now, whether you’re on a Xeon or an H100.</p> Naively running llama-cli</code> on a DDR3 machine without a GPU is horrendously slow, even if it can run it, because it’s optimized for a generic GPU usecase, and often leaves a lot of improvements on the table. Further, it simply doesn’t have most of the actual optimizations that the state of the art currently uses to run these at scale.</p> The remedy is to pull every optimization lever ik_llama.cpp</code> exposes. Most of them are slightly obscure.</p> Here is the magic spell that makes this actually run.</p> llama-cli</span></span> \ </span></span></span> --</span>model</span> gemma-4-26B-A4B-it-Q8_0.gguf \ </span></span></span> --</span>model-draft</span> gemma-4-26B-A4B-it-assistant-GGUF/\ </span></span></span>wikitext-2-raw_ik-llama-mtp_drafter-conservative/\ </span></span></span>gemma-4-26B-A4B-it-assistant-Q8_0.gguf \ </span></span></span> --</span>spec-type</span> mtp --</span>draft-max</span> 3 --</span>draft-p-min</span> 0.0 --</span>spec-autotune</span> \ </span></span></span> -</span>cnv</span> --</span>color</span> --</span>jinja</span> --</span>special</span> \ </span></span></span> -</span>sm</span> graph -</span>smgs</span> -</span>sas</span> -</span>mea</span> 256 --</span>split-mode-f32</span> \ </span></span></span> --</span>temp</span> 0.7 -</span>t</span> 8 --</span>parallel</span> 8 \ </span></span></span> --</span>cpu-moe</span> --</span>merge-up-gate-experts</span> \ </span></span></span> --</span>flash-attn</span> on --</span>mla-use</span> 3 \ </span></span></span> --</span>mlock</span> --</span>run-time-repack</span> --</span>no-kv-offload</span></span> </span></code></pre> Under a blackbox tool like ollama</code> you never see this line. On aging hardware you have to understand what each flag does, because half of them won’t take, and the engine will tell you so in passing.</p> Speculative decoding.</strong></p> --spec-type mtp --draft-max 3 --draft-p-min 0.0 --spec-autotune </span></code></pre> This pairs the 26B verifier with the small drafter from the previous post. Up to three tokens per draft (--draft-max 3</code>), all probabilities accepted (--draft-p-min 0.0</code>), --spec-autotune</code> adjusting the chain length per workload.</p> This ties directly back to our previous discussion about the memory-bound decoder pass.</p> When a model uses a long reasoning chain, it is generating those “thinking” tokens one by one. Even if the internal reasoning is hidden from the user and all you see is a short final answer, the hardware still has to perform a full decoder pass for every single token in that hidden chain.</p> In fact, speculative decoding is currently one of the most brilliant software workarounds the AI industry has invented to bypass the “memory wall,” and spec autotune is how you squeeze the maximum speed out of it.</p> The argument for speculative decoding is stronger on CPU than on GPU. CPU compute is cheap relative to the cost of streaming the verifier’s weights through cache, so spending extra cycles on a tiny drafter whose active layers easily fit in L3 buys tokens at very little marginal cost. The drafter’s working set fits in L3. The verifier however spills out of everything.</p> CPU and MoE routing.</strong></p> --cpu-moe --merge-up-gate-experts -t 8 --parallel 8 </span></code></pre> Gemma 4 26B-A4B has 128 experts with 8 active per token, giving about 3.8B active parameters out of ~25.2B total. --cpu-moe</code> tunes the routing for CPU cache hierarchies.</p> CPUs handle memory very differently than GPUs. While a GPU has a massive pool of ultra-fast High-Bandwidth Memory (HBM), a CPU relies on small, lightning-fast “caches” (L1, L2, L3) built directly onto the processor chip.</p> In an MoE model, constantly jumping around between 128 different experts can cause “cache thrashing”, where the CPU constantly has to dump its cache and fetch new weights from the much slower main system RAM (normally DDR4/DDR5, we’re on DDR3!).</p> This flag tells the router to be smarter about how it picks experts, optimizing the sequence so the weights stay neatly inside the CPU’s local cache for as long as possible.</p> --merge-up-gate-experts</code> fuses two per-expert projections into a single matmul, which the logs confirm:</p> fused_up_gate = 1 </span></code></pre> This is a software trick to bypass the memory bandwidth bottleneck we discussed earlier.</p> Inside the experts, the math operations require data to be passed through different layers. Normally, the processor would calculate an “up projection”, write the result to memory, then load the weights for a “gate projection”, calculate that, and combine them. That requires moving data across the memory bus multiple times.</p> Instead of doing two separate trips over the memory bus, it combines the operations into a single step.</p> -t 8</code> matches physical cores. The machine has 16 SMT threads but only 8 cores. On a memory-bound workload, oversubscribing threads adds scheduling cost without adding throughput: the cores are waiting on DDR3, not on each other.</p> Memory pinning, repacking, KV cache.</strong></p> --mlock --run-time-repack --no-kv-offload </span></code></pre> --run-time-repack</code> reorganizes weight matrices in memory immediately before inference to match the CPU’s cache layout. The logs confirm:</p> ============ Repacked 265 tensors </span></code></pre> Processors have their own ultra-fast, built-in memory called caches (L1, L2, and L3). However, these caches expect data to be fed to them in very specific shapes and sizes.</p> If the AI’s weight matrices are sitting in system RAM in a generic layout, the CPU has to awkwardly pull the data in pieces, resulting in “cache misses” where the CPU stalls. --run-time-repack</code> tells the engine to spend a few seconds during startup to physically reorganize the massive tables of numbers in the RAM so they perfectly align with how the CPU wants to ingest them. It pays a small time penalty upfront to guarantee maximum memory bandwidth during the actual text generation.</p> --mlock</code> is meant to pin the model in RAM so the OS cannot swap any of it to disk.</p> mlock</code> stands for “memory lock”, suprising, I know! In standard operating systems, if the system starts running out of RAM, it will quietly take data that hasn’t been used in a few seconds and “swap” (or page) it to the physical hard drive.</p> If an OS tries to swap out 27GB of AI weights to a disk, the generation speed will instantly drop to zero while the system chokes trying to read it back. --mlock</code> tells the Linux kernel: “Pin this 27GB strictly in physical RAM. Do not ever move it to the disk.”</p> Notice that if you’re not careful, you’ll see this:</p> warning: failed to mlock 27628376064-byte buffer </span> (after previously locking 0 bytes): Cannot allocate memory </span>Try increasing RLIMIT_MEMLOCK ('ulimit -l' as root). </span></code></pre> The flag is fine; the kernel-side memlock limit isn’t set high enough to pin a 27 GB buffer. This is not an LLM-shaped problem at all — it’s a ulimit default — and it’s the kind of footgun the blackbox tools paper over by simply not asking for the optimization in the first place.</p> Consider that for a moment, that many tools by default will just have no problem putting your model into swap if it decided that’s the best option. You can imagine how much this can hurt performance…</p> --no-kv-offload</code> tells the engine not to look for a GPU for the KV cache. There isn’t one to find, but the flag short-circuits the check.</p> The KV (Key-Value) cache is the AI’s short-term memory — it stores the context of the current conversation so the model doesn’t have to re-read the entire prompt for every new token.</p> Because the KV cache is constantly being read from and written to, AI engines usually try to “offload” it to a GPU, which has much faster memory than we do.</p> Since this specific setup is highly optimized to run purely on a CPU, letting the engine search the hardware buses for a GPU that doesn’t exist is a waste of time and could throw an error. This flag explicitly short-circuits that check, telling the engine to just keep the short-term memory in the system RAM alongside the weights.</p> Graph layout.</strong></p> I’ve tried my best to keep this easy to understand, but this part is just plain hard to make explain in a single blog post.</p> </blockquote> Now onto dark arts. A common frustration in bleeding-edge AI software is that the engine is being developed so fast that the developers don’t have time to write official documentation. If you want to know how to optimize the engine, you have to dig through the raw code or read the Github Pull Request (PR) comments between the developers.</p> -sm graph -smgs -sas -mea 256 --split-mode-f32 </span></code></pre> These flags govern how the computational graph is allocated across memory regions. The full documentation ultimatley lives in the code, even if it has some documentation.</p> The flag -sm graph</code> tells the engine to use Split Mode in the Graph mode (often known in the industry as Tensor Parallelism). This is entirely about how you divide the massive math workload across multiple processors or memory regions (like multiple CPU sockets or GPUs).</p> Layer Split (The Default/Fallback): The engine slices the model horizontally. Processor A calculates Layers 1–10, then sends the data over the system bus to Processor B, which calculates Layers 11–20. While Processor A is working, Processor B is sitting idle.</p> </li> Graph Split (The Goal): The engine slices the computational graph vertically. Processor A and Processor B calculate different halves of Layer 1 at the exact same time, combine their answers, and move to Layer 2 together. This keeps all hardware running at 100% simultaneously, drastically improving generation speed.</p> </li> </ul> On this run, the engine declines:</p> ======================================================= </span>Split mode 'graph' is not supported for Gemma4 external MTP </span> => changing split mode to 'layer' </span>======================================================= </span></code></pre> Because MTP creates a much more complicated web of math at the very end of the network, this inference engine simply hasn’t gotten support yet to safely “graph split” (vertically slice) an MTP architecture yet. When the engine boots up, it detects the MTP layers, realizes -sm graph will break the math, and safely downgrades to the slower, sequential layer split so the model can still run.</p> I’ve included it because it will likely be very helpful in the future, so you should try your luck if you’re working on a newer version.</p> While -sm graph</code> was disabled, these other flags still apply to how the engine manages memory:</p> -sas</code> (Split Across Sockets): Explicitly tells the engine how to divide the workload across different physical CPU sockets (NUMA nodes) on a server motherboard. You may note we only have one CPU, but we could get more later, it’s a nice optimization, just bench it to be safe if you do this, since older boards may break current day assumptions.</p> </li> --split-mode-f32</code>: When data is split across processors, it has to be stitched back together. This flag forces those intermediate connection points to use 32-bit floating-point precision (higher quality math). It prevents the AI from losing intelligence or hallucinating due to rounding errors during the split.</p> </li> </ul> And don’t worry if you see this:</p> Oops: tensor with strange name rope_freqs.weight </span></code></pre> It has a strange name. Strange names will not stop us here. :D</p> Attention.</strong></p> Look. ikawrakow</code>, creator of ik_llama.cpp is beyond the word “craked”.</p> Kawrakow wrote custom CPU kernels to handle Flash Attention, bypassing the need for a GPU during heavy context processing.</p> This let’s us do something that normally you only do on a GPU.</p> --flash-attn on --mla-use 3 </span></code></pre> Flash Attention fuses the attention softmax with its matmuls to avoid materializing the full attention matrix. Duh, anyone knows this, but I’ll try to explain it.</p> To generate text, an AI has to calculate how every single word in your prompt relates to every other word. Mathematically, this creates a grid of size N×N (where N is the number of tokens).</p> If you give the AI a short sentence, that grid is small. But if you feed it a 100,000-word document, that matrix explodes into 10 billion cells. Normally, the processor calculates this massive matrix and “materializes” it — meaning it physically writes the entire giant grid out to the main system RAM, only to immediately read it back for the next step.</p> Flash Attention applies the Kernel Fusion trick, but to the attention mechanism. It calculates the attention scores in small chunks and fuses the math (the softmax) so that the giant N×N matrix is never actually written to RAM. It is calculated and consumed entirely inside the processor’s ultra-fast local cache.</p> Flash Attention was originally invented strictly for GPUs because it relies on how GPU hardware handles memory blocks. Successfully porting this highly complex, hardware-specific optimization to work on standard CPUs is a massive software engineering achievement.</strong> Well done ikawrakow</a>.</p> --mla-use 3</code> enables Multi-Head Latent Attention. Earlier, we discussed the KV Cache (the AI’s short-term memory of the conversation that prevents it from having to re-read the whole prompt for every word).</p> In standard architectures, storing the raw Key and Value data for every single token eats up RAM incredibly fast. Multi-Head Latent Attention (MLA) is a breakthrough architecture that heavily compresses this short-term memory. Instead of saving raw data for every token, it compresses the Keys and Values into a much smaller, dense mathematical representation (a “latent” space).</p> This drastically reduces the memory footprint of the KV cache, allowing the model to remember massive conversations without running out of system RAM. The flag --mla-use 3</code> simply tells the engine to activate a specific tier or kernel implementation of this compression.</p> But all of this is just experimental stuff right, like the split mode graph? Nah. The logs confirm both took:</p> flash_attn = 1 </span>fused_moe = 1 </span>fused_up_gate = 1 </span></code></pre> The memory accounting from the logs:</p> ------------------- Layer sizes: </span>Layer 0: 825.98, 2048.00, 2873.98 77.00 MiB </span>... </span>Layer 29: 840.59, 1024.00, 1864.59 77.00 MiB </span>Layer 30: 748.00, 435.00, 1183.00 MiB (output layer) </span>-------------------------------------------------------------------------- </span>Total : 24852.46, 56755.00, 81607.46 MiB </span>Memory required for model tensors + cache: 82355 MiB </span></code></pre> An 82 GB footprint in DDR3 on a 2016 Xeon. About 25 GB of weights and 56 GB of KV cache at the full 262K context. The KV cache is larger than the model.</p> That a working configuration requires 25 flags, half of which are undocumented and a quarter of which fail silently, is a reasonable working definition of the usability moat described in the first post</a>.</p> The engine loads a 25B-parameter MoE, runs speculative decoding against an MTP drafter, and generates text at reading speed on hardware that was old when the architecture in question hadn’t been invented yet.</p> When we started this series a week ago, the state of local open-weights AI looked grim. We began by pulling back the curtain on the industry’s favorite marketing spin: the idea that dropping a massive, uncalibrated weights file onto a repository constitutes “open source.” We looked at the massive usability moat built out of missing documentation, silent defaults, and black-box wrappers that hide performance-killing decisions under the guise of user-friendliness.</p> In the second post, we rolled up our sleeves and waded into the muck. We hunted down obscure, unmerged pull requests, compiled specialized forks (ik_llama.cpp), flipped the standard logic of quantization on its head to build highly precise speculative decoding drafters, and wrote custom scripts to scrub infrastructure data leaks out of our GGUF metadata.</p> Finally, in this post, we put our money where our mouth is. We dragged a 2016 enterprise relic out of the closet — NAY, out of the grave, a single Intel Xeon running on agonizingly slow DDR3 RAM with absolutely no GPU to speak of — and forced it to run a cutting-edge, 26-billion-parameter Mixture-of-Experts architecture at reading speed. We did without throwing exotic hardware at the problem. Instead we treated the deployment pipeline as a serious thing, and mapped the architecture directly to physical hardware, tuning memory allocation, and unlocking the absolute limits of CPU cache optimization.</p> The lesson here is simple: The bottleneck to running state-of-the-art AI locally isn’t just in the silicon. It’s the need to understand how the inferrence engine actually works. Deeply.</p> While a cluster of data-center graphics cards, a corporate API token, or a massive budget are all extremly useful for specific workloads, for the ones that the open models cover, you just need refurbed hardware and to refuse to let black-box tools hold the steering wheel. Armed with the right fork, calibrated quants, and an understanding of the memory architecture under your hood, the usability moat vanishes.</p> The bleeding edge of Open Weight AI isn’t locked behind a paywall or a model proivider. If you’re already running a homelab, It’s sitting right there on the command line of a ten-year-old server.</p> Welcome to the other side of the moat. Now go download the quants</a> and get your hands dirty.</p> Thanks for reading :D</p> Quantizing Gemma 4's MTP drafters Christina Sørensen — Fri, 29 May 2026 00:00:00 +0000 A while after Gemma 4 came out, Google released a set of Multi-Token-Prediction (MTP) drafters for it. Pairing a large model (the verifier</em>) with a tiny MTP drafter (the assistant</em>) enables speculative decoding: the small model guesses the next few tokens, and the big model verifies them in one parallel pass. When it works, the throughput gain is most of the way to free.</p> To put it shortly: MTPs are one of the rare free lunches we get with LLMs. Or well, on paper, like with free lunches, that’s the assumption.</p> These are notes on getting that working locally. The previous post</a> was about why this should have been easier than it was.</p> If you try to convert the Gemma 4 MTP drafter weights using mainline ggml-org/llama.cpp</code>, the converter rejects them:</p> Model Gemma4AssistantForCausalLM is not supported </span></code></pre> Mainline supports the base</em> Gemma 4 models, but it does not know what a gemma4_assistant</code> is. The converter lacks the class, and the runtime lacks the MTP API. The runtime PR #18886</a> has been in draft for months. A companion converter PR #22738</a> was opened and closed unmerged with one of four tasks done.</p> The working path is a fork: ikawrakow/ik_llama.cpp</code></a>, specifically the feat/gemma-4-mtp</code> branch from PR #1744</a>. The resulting GGUFs only load in that fork. Downstream wrappers — Ollama, LM Studio, jan.ai, llama-cpp-python</code> — will all reject them until they backport the changes.</p> This is the cost of running new architectures the week they ship: you swap inference backends to match the model, not the other way around.</p> Drafters are tiny. The E2B drafter’s MTP head is 78M parameters; the 31B’s drafter is roughly 470M; the 26B-A4B’s is about 408M.</p> Their job is to predict tokens accurately enough that the verifier accepts</em> them, so acceptance rate dominates everything else. Aggressive quantization causes the verifier to reject more drafts, and the pipeline becomes slower than running the verifier alone.</p> The quantization policy for drafters is the inverse of what you’d do for a large base model:</p> Do use:</strong> bf16</code>, f16</code>, Q8_0</code>, Q6_K</code>, Q5_K_M</code>, Q4_K_M</code>.</li> Do not use:</strong> IQ1_</code>, IQ2_</code>, Q2_K</code>, Q3_</code>. These cost more rejected drafts than they save in load time.</li> </ul> imatrix</code> calibration is not available for these drafters at the moment. PR #1744 builds the gemma4_mtp</code> drafter graph with a hardcoded GGML_ASSERT(has_target_ctx)</code>, so standalone llama-imatrix</code> runs abort in llama_decode</code> before producing anything. The IK fork’s signature quants (IQ4_KS</code>, IQ4_KSS</code>, IQ5_KS</code>) derive their precision-per-bit advantage from imatrix-guided scale selection, so without an imatrix they collapse to roughly K-quant quality at the same bit budget. Shipping them under those labels would be misleading.</p> Quantizing directly from bf16</code>/f16</code> to Q4_K_M</code>/Q8_0</code> is the realistic option until the tooling catches up. At ≥4 bits the precision gap from missing the imatrix is small — single-digit percent on most benchmarks. At Q3 and below it would matter, which is the other reason Q3 isn’t a sensible target here.</p> One empirical surprise: on a 4060 Laptop GPU, pairing unsloth’s Gemma-4-E4B-it-Q4_K_M.gguf</code> verifier against both a Q8_0</code> and a Q4_K_M</code> drafter at --draft-max 3</code>, the Q4_K_M</code> drafter beat the Q8_0</code> by ~13% in total throughput. The smaller drafter’s faster draft step outweighed the acceptance-rate cost. The “always pick the highest-bit drafter” rule assumes you are memory-bandwidth-rich and compute-poor; on a laptop GPU you are neither.</p> Benchmark your own silicon.</p> llama-quantize</code> silently embeds the absolute on-disk paths of your imatrix file and calibration corpus into the GGUF metadata. There is no flag to disable this at quantize time. The -f16.gguf</code> intermediate is clean; only quantized files have the leak.</p> Inspect any quantized GGUF and the two offending keys are sitting there:</p> nix-shell</span></span> -</span>p</span> python3Packages.gguf --</span>run</span> \ </span></span></span> '</span>gguf-dump --no-tensors model.gguf'</span></span></span> </span></code></pre> Key</th> What it leaks</th></tr></thead> quantize.imatrix.file</code></td> Absolute path to the imatrix file on the build host</td></tr> quantize.imatrix.dataset</code></td> Absolute path (and filename) of the calibration corpus</td></tr> </tbody></table> Uploading the raw output of a quantization pipeline publishes your infrastructure layout — usernames, ZFS pool names, directory structure — alongside the weights.</p> The fix is gguf_new_metadata</code> from the Python gguf</code> package, which rewrites the file with selected keys removed without touching the tensors:</p> nix-shell</span></span> -</span>p</span> python3Packages.gguf --</span>run</span> '</span> </span></span></span> for f in /srv/models/<MODEL>-GGUF/.gguf; do </span></span></span> [ "$f" = "-f16.gguf" ] && continue </span></span></span> python3 -m gguf.scripts.gguf_new_metadata \ </span></span></span> --remove-metadata quantize.imatrix.file \ </span></span></span> --remove-metadata quantize.imatrix.dataset \ </span></span></span> --force \ </span></span></span> "$f" "$f.clean" </span></span></span> done </span></span></span>'</span></span></span> </span></code></pre> Each rewrite runs at ~5–15s per GiB and shrinks the file by exactly the size of the two strings removed (≈190 bytes).</p> Verify before uploading:</p> gguf-dump</span></span> --</span>no-tensors</span> model-clean.gguf</span> \|</span> grep</span></span> -</span>v</span> INFO</span> </span></code></pre> kv_count</code> should be exactly 2 less than the original. The imatrix._count</code> integer fields stay (those carry no path information). A smoke test confirms the rewritten file still loads:</p> llama-completion</span></span> -</span>m</span> model-clean.gguf --</span>jinja</span> \ </span></span></span> -</span>p</span> '</span>The capital of France is'</span></span> -</span>n</span> 20 --</span>temp</span> 0</span> </span></code></pre> Coherent output means the rewrite didn’t corrupt anything.</p> Sanitized quants in hand, the next step is pairing. A drafter is not a standalone language model: it has to be paired with a verifier that shares its exact vocabulary. For Gemma 4 that vocabulary is 262,144 tokens.</p> The size of that vocabulary is also the reason the MTP speedup applies to structured output. The drafter sees the same <\|tool_call\|></code>, <\|tool_response\|></code>, and <\|channel\|></code> tokens as the verifier, so tool-calling agents and structured-generation workflows benefit on the full response, not just the natural-language parts. The vocabulary itself isn’t quantized — it lives in a separate GGUF section.</p> A working ik_llama.cpp</code> invocation:</p> llama-server</span></span> \ </span></span></span> --</span>model</span> gemma-4-31B-it-Q8_0.gguf \ </span></span></span> --</span>model-draft</span> gemma-4-31B-it-assistant-Q8_0.gguf \ </span></span></span> --</span>spec-type</span> mtp \ </span></span></span> --</span>draft-max</span> 3 \ </span></span></span> --</span>draft-p-min</span> 0.0 \ </span></span></span> -</span>ngld</span> 99 \ </span></span></span> --</span>n-gpu-layers</span> 99 \ </span></span></span> --</span>ctx-size</span> 32768 \ </span></span></span> -</span>ctk</span> q8_0 -</span>ctv</span> q8_0 \ </span></span></span> -</span>b</span> 1024 -</span>ub</span> 1024 \ </span></span></span> --</span>jinja</span></span> </span></code></pre> The relevant levers:</p> --spec-type mtp</code> enables the speculative-decoding path from PR #1744.</li> --model-draft</code> is the drafter GGUF; its vocabulary must match the verifier.</li> --draft-max 3</code> sets the speculative chain length. Values from 1 to 4 are all reasonable; --spec-autotune</code> will probe and pick one for your workload.</li> --draft-p-min 0.0</code> accepts all drafts; raising it shortens chains on difficult prompts.</li> --jinja</code> applies the model’s chat template, including Gemma 4’s tool-call format.</li> </ul> PR #1744’s reference numbers, 31B verifier + 31B drafter at Q8_0/Q8_0 on a data-center GPU:</p> Run</th> Throughput</th> Acceptance</th></tr></thead> Baseline (no MTP)</td> ~21 t/s</td> —</td></tr> MTP --draft-max 1</code></td> ~35 t/s</td> ~89%</td></tr> MTP --draft-max 2</code></td> ~44 t/s</td> ~83%</td></tr> MTP --draft-max 3</code></td> ~49 t/s</td> ~74%</td></tr> MTP --draft-max 4</code></td> ~49 t/s</td> ~64%</td></tr> </tbody></table> A bit more than 2× throughput, after all of the above.</p> The throughput gains are real, and on the right hardware they are substantial. The work it takes to surface them is also real: undocumented CLI divergences, an obscure fork, manual metadata sanitization, quantization rules that invert the usual advice. None of this is in the model card or the upstream README. The path to a working setup is reconstructed from PR threads and stack traces.</p> The next post</a> in this series takes the result and runs it on a 2016 Xeon with no GPU, which is where the command line stops being a paragraph.</p> Open weights are not open source Christina Sørensen — Tue, 26 May 2026 00:00:00 +0000 I’ve been quantizing Gemma 4 and it’s MTP drafters recently. The next two posts are about how that went. This one is about why it was so much work.</p> When a lab releases a model under an open license, what they almost always release is the weights. Sometimes a tokenizer. Occasionally a short technical report. Rarely the training data, data-processing code, training code, evaluation harness, or the recipes used to produce the published quantizations. People still call this open source, and I think that’s misleading.</p> If you can’t rebuild it, you can’t fork it. If you can’t fork it, the word doesn’t mean what it used to.</p> Calling it open is like calling an executable binary open because you can execute it. Unless you consider assembly code open source, I think it’s really obvious why this misses the mark, and is exellent marketing spin.</p> The people releasing models don’t seem to feel responsible for whether anyone can run them, or have good results. Its more like box ticking. Here it is?</p> There is no shared vocabulary for how a model should be packaged, or which inference engine it expects, or what quantization recipe was used to make the file on whatever hub they pick. The settings that actually decide whether a model is usable on a given machine — mlock</code>, KV-cache dtype, flash attention, MoE routing, draft model pairing — live in PR descriptions and Discord screenshots. The model card lists the benchmark scores and links a HuggingFace space.</p> So you download the weights, get a result that’s slow or incoherent or both, and you don’t know whether the model is bad, your quantization is bad, your runtime is bad, or you’ve just configured something wrong. Most people, reasonably, conclude it’s just their hardware, or that open models don’t actually work for normal users.</p> The tools that have grown up to paper over this make the problem worse, not better. Ollama is the obvious example. It picks a quantization, runtime, context and batch sizes for you, and doesn’t tell you what it picked or why. When the output is bad, there’s no thread to pull. You can’t tell if you’re holding it wrong because the thing was designed so you can’t hold it at all. It’s just a blackbox!</p> This wouldn’t matter much if the defaults were good. They aren’t. A default Q4_0 of a small model with no imatrix calibration and a mismatched chat template will be much worse than the same model run carefully, and the user has no way of knowing the difference exists.</p> AND NO ONE REALLY TALKS ABOUT HOW MUCH THIS MATTERS!</p> Concretely: a while after Gemma 4 came out, Google released a set of Multi-Token-Prediction drafters for it, designed to make the base models faster via speculative decoding. The obvious thing to do is download the GGUF and run it under llama.cpp</code>.</p> That doesn’t work, and the reasons it doesn’t work are a fair sample of the genre:</p> Mainline llama.cpp</code> doesn’t know what a Gemma4AssistantForCausalLM</code> is. The converter rejects the drafter outright. To convert it, you have to find ikawrakow/ik_llama.cpp</code>, on a feature branch, from a PR that hasn’t merged.</li> llama-imatrix</code> aborts on that branch’s MTP graph because the build hardcodes an assertion that a target context is bound. So you skip imatrix calibration, and your Q4 quants are noticeably worse than the ones every other model on the hub ships with.</li> llama-quantize</code> writes the absolute on-disk paths of your imatrix file and calibration corpus into the GGUF metadata. There’s no flag to turn this off. If you upload without scrubbing, you publish your home directory layout next to the weights.</li> </ul> None of this is in the model card. None of it is in the upstream README. You find it by trying things and reading stack traces.</p> No one does this. Seriously. I mean yes. Part of it is that they just use what the people that did bother to do it made, fair. But there’s like 4 quants for this on huggingface at the time of writing, one is mine. FOUR! For the state of the art MTP/MOE models!</p> The hardware to run a 30B-class MoE at reading speed isn’t exotic. A five-year-old workstation with enough RAM will do it. The bottleneck isn’t the silicon, it’s everything between the weights file and a working configuration: which fork, which quant, which flags, which template, which version of which template. Ohh, and getting it all running, good luck unless you’re… literally an infrastructure engineer? Most people give up somewhere in the middle and go back to the API, and I don’t blame them.</p> I do think the labs releasing these models could make this much less bad than they have. Publishing the imatrix corpus would help. Publishing the quantization recipe would help. Picking one reference runtime and keeping the model working on it would help. Treating the deployment story as part of the release, rather than something the community is expected to reconstruct, would help.</p> Until that changes, “open source” for language models mostly means “someone else has to figure out how to run it, and none of the freedoms that used to mean open source are actually present”, and the people best positioned to figure it out are charging by the token.</p> I’ll write up the actual quantization steps — which flags, which traps, what the metadata leak looks like, how to scrub it — and what it took to run the result on a 2016 Xeon with no GPU, in two follow-up posts this week. Thanks for reading :)</p> Another NixOS Election Cycle Christina Sørensen — Thu, 02 Oct 2025 00:00:00 +0000 And here I thought we’d have a boring election cycle</h2> If you’ve not been following NixOS governance, recently the moderation team decided to all collectively resign. I’ve been informed that I’ve been one of the people that certain steering committee members have wanted to moderate harder, specifically because I’ve been calling out their undisclosed conflicts of interest, and providing actual level headed, direct criticisms that they’ve yet managed to answer.</p> This is despite their alleged pledge to respond to community concerns. One of the people I’ve been led to believe specifically has been pushing for aggressively moderating me has yet to respond to any messages I’ve sent them in private, which to me is even more odd, they’re not rude, and could likely have cleared up any legitimate moderation issues.</p> I think that, looking back at the previous steering committee, and even watching their talks at NixCon, it was quite clear that most of them were not really interested in governance. Paraphrasing, most of them said a version of “I’m not the most competent person to be here”, and those were the ones that at least have the insight to make that realization.</p> I think that many were there simply to add a new line to their resume.</p> What does it mean for me running?</h2> Nothing. I’ll run. If enough people vote for me, I’ll do what I set out to do in my candidate statement. If I’m not elected, I’ll put my focus elsewhere, and make myself available to any elected Steering Committee members on policy and governance questions, just like the last cycle.</p> I don’t think that the Steering Committee is the only (or even the most effective) way for me to improve the project and ecosystem. But it is an option I’d be foolish to throw away. And I have much I want to do.</p> Here is the start.</p> Process</em></h4> I’ve read the Steering Committee retrospective from Gabriella</a>. To me, the challenges raised here are of primary concern.</p> Currently, the steering committee isn’t leading the community. And if the final instance required for decisive action in the project is indescisive, then what’s the point?</p> So that’s my primary concern. The steering committee should actually get things done.</p> I think we can solve this.</p> Transparency</em></h4> Meeting notes and employers of board and steering committee memeber should be made public - to the extend possible. There should be public logs of both.</p> Everything should default to being documented in public, including decisions and how members vote.</p> If the steering committee finds certain subjects too sensitive, they may be redacted, but even then, the fact that a vote or discussion took place should be documented. Who voted for and against - even if we can’t say for what - should be written down for everyone to see.</p> The steering committee must become auditable and accountable to everyone in the project.</p> Unblocking Adoption</em></h4> Flakes aren’t unstable. That’s mainly a naming convention at this point. It is a major disadvantage for the project, and the official installer adds friction.</p> Flakes should be installed by default. Defaults should be sane.</p> Many decisions in the project aren’t made based on what’s best for the project, but where measures to alleviate immediate harm. Now those same measures have themselves become harmful.</p> By engaging with the community, and listening for places where their colleagues are blocked or choosing alternatives to official tools, I’ll correct any blockers with the steering committee and community, and ensure we become the best choice.</p> I think Nix is a paradigm shift in how we do computers, and I want to convince the rest of the world of that. I want to work with Nix for the rest of my career, and for that to become a reality, I’m running for the steering committee again.</p> A final question: Will you go the distance</h2> I know a lot of people are scared that they’ll end up voting for someone that will turn passive and be checked out. I am scared of voting for such people. It was a major pain in the last election, that even if I was aligned with a candidate and they got elected, they just didn’t actually push for anything, and seemed to essentially give up.</p> If that’s something you’re scared will happen, I think specially if you’re a long time community member, the best proof I can give that it wont happen if you vote for me is that… I’m still here. I’d not want to waste your vote.</p> But I’d also like to temper your expectations. I can only do so much. If you vote for a bunch of people that are not gonna go the distance, then my primary role becomes keeping you updated on why the process remains broken. I can be a voice for accountability, but I cannot be the entire committee.</p> Tracking Sattelites with Consumer Components Christina Sørensen — Sun, 05 Jan 2025 06:00:00 +0000 A Global Network of Ground Stations</h2> At 38c3</a> I attended a workshop on building a sattelite ground stations for communicating with LoRa based sattelites, typically used for research or weather probes.</p> Generally speaking, these are so called “cubesats”</a>, which are small, relatively</em> inexpensive sattelites usually built up of off the shelf components in a modular fashion. In fact, it turns out that with one of these inexpensive sattelites, a small launch would actually only</em> cost in the range of $50,000-$100,000, not a small amount, but still, a lot more accessible than what you would initially imagine. Combine this with LoRa (literally Lo</strong>ng Ra</strong>nge) — a proprietary radio communication technique — and you have the components for a globe spanning network of sattelites and amatur ground stations.</p> </p> This has lead to a suprising amount of these sattelites currently in low-earth orbit around the globe</a>. Typically these aren’t geostationary (meaning located over the same point on the planet at all times), and so, it is useful for e.g. a smaller university to have access to a globe spanning network of sattelite stations.</p> TinyGS</h2> Enter TinyGS, tiny ground stations</a>. TinyGS is an open network of Ground stations distributed across the globe, run by amateurs. It allows operating LoRa based weather probes, sattelites, and other “flying objects”, using consumer electronics.</p> The basic idea is that you get an ESP32 as well as a some sort of LoRa module, an antenna, and flash the tinygs firmware onto it. Then you just connect it to the tinygs servers using MQTT, and you will be able to opearte your board on the tinygs website, as well as being able to contribute to the global sattelite tracking effort.</p> </p> Build your own Ground Station</h2> Hardware</h3> If you wanna build your own, the easiest possible route is just to acquire a LoRa32 V2.1_1.6 - 433MHZ CH9102F</code> board. Usually you can find these online, and make sure that you get one with a small antenna.</p> </p> You’ll need to flash the firmware, the easiest way is to use the TinyGS web flashing tool</a>, and sorry, but you’ll need to use a chrome based browser for this.</p> Setup</h3> What’s worse, you’ll also need to install telegram, and create an account, as the TinyGS projects communications mainly happen there, and that means the registration bot for your new board also lives on telegram.</p> Join the TinyGS telegram group here</a>, and then start a conversation with the tinygs personal bot here</a>. Also, make sure you join the TinyGS community chat in the group first, as the bot will require you to be a member before signing you up. When you are in that chat, send the /start</code> command to the bot for some info about it’s usage. You’re mainly just interested in the /mqtt</code> command, which gives you your credentials for the mqtt server.</p> </p> Now with your credentials, and your participation in the TinyGS community (hi!), you’ll want to plug in your flashed board. After a few moments, it should start it’s own WiFi AP, that you can connect to. On the screen of your board, it should show the IP address that you’ll then want to connect to on the boards WiFi. This will give you a configuration inteface.</p> </p> Here, you enter the configuration, and then configure a ground station name, create an admin password (to be used with the admin</code> account), enter the WiFI SSID and Password that you want the board to use for internet connectivity, your lattitude and longitude (this doesn’t have to be exact</em>, but should be vaguely correct withing 25km~ because it is used to determine what sattelites to try and connect to), and a timezone. Further, you’ll want to add your MQTT credentials from the telegram bot.</p> If you’ve done everything correctly, you’ll receive a message on telegram from the bot, like I did in the above screenshot. You’re now on the network.</p> Start operating your sattelite</h3> After you’ve setup your station, you’ll want to /login</code> with the bot to be able to operate it fully.</p> </p> This will give you access to even more information about your device, and the ability to edit it’s description and image, and to control how it is tracking sattelites, or setup manual tracking if you wanna just track something specific, or you don’t like the automatic selection.</p> </p> Below that, you’ll find a log of the packets you have recieved so far.</p> </p> Notice here that the red circle indicated a CRC checksum error, meaning you recieved a faulty package. Also notice that there are two packets with a CRC error that were caught by just one station (meaning just your station). In this case, if you click on the packet, you’ll get the text that:</p> Unrecognized packet. This packet does not match the known structure of this satellite. It might be an unknown packet or just terrestrial noise.</p> </blockquote> However, for those that don’t have a CRC error, that’s a real sattelite packet you’ve caught. Well done!. If you click on it you can even read the details about what you caught.</p> For example, the HOD-HOD-1A</code> capture from above provides the following information when you click on it.</p> </p> As you can see, it shows what the content of the packet was, both as text, hexadecimal, and parsed json that you can expand and browse through. Below that, you’ll see a list of other stations that caught the packet. The view to the side shows the geographical location of the stations that caught the sattelite communication.</p> I wanna capture more packets!</h3> As you will likely realize after running your station for a few days, you’re probably not capturing a lot of packets. This is because the default antenna doesn’t really perform well, and likely, you’ll also ideally want to make sure that your station is outside, and isn’t obstructed, and that weather conditions are favorable (anecdotally, I also tend to capture much more packets at night).</p> I’ll not elaborate more on this in this post, as it is something I’m working on myself, but I may follow up later with a separate post on what it takes to increase the amount of packets you capture. In short, you’ll want a 1/4-wave ground plane antenna.</p> Obligatory Conclusion</h2> With inexpensive consumer hardware, it is possible to build your own sattelite ground station, and join a globe spanning network of space enthusiasts capturing sattelite data.</p> That is pretty cool :3</p> Conventional Commits Change my Brain Christina Sørensen — Wed, 24 Jul 2024 00:00:00 +0000 What is a conventional commit?</h2> In case you’re not aware, conventional commits</a> is a standard format for writing commit summaries. It looks like this.</p> build(deps): bump git2 from 0.18.3 to 0.19.0 </span> </span>Bumps [git2](https://github.com/rust-lang/git2-rs) from 0.18.3 to 0.19.0. </span>- [Changelog](https://github.com/rust-lang/git2-rs/blob/master/CHANGELOG.md) </span>- [Commits](https://github.com/rust-lang/git2-rs/compare/git2-0.18.3...git2-0.19.0) </span> </span>--- </span>updated-dependencies: </span>- dependency-name: git2 </span> dependency-type: direct:production </span> update-type: version-update:semver-minor </span>... </span> </span>Signed-off-by: dependabot[bot] <support@github.com> </span></code></pre> This example is from dependabot</a>, which does conform to conventional commits. Notice that only the first line, the commit summary</em>, is part of the conventional commits spec here. The last line is a “footer”[1]</a></sup>, or technically, a trailer</em></a> added by the git commit -s</code> flag, -s</code> short for --signoff</code>.</p> The exact spec looks like this:</p> <type>[optional scope]: <description> </span> </span>[optional body] </span> </span>[optional footer(s)] </span></code></pre> Unlike many project specific standards, such as that of angular</a> or nixpkgs</a>, what sets conventional commits apart is that it aims to be a universal commit standard, a bit like how semantic versioning</a> seeks to be a universal software versioning standard.</p> In the same sense as semantic versioning provides meaning through version numbers, such as a major</code> number increasing meaning a breaking change, conventional commits also seeks to encode information about the state of a project in the commit summaries, and provides enough information to reasonably tell when it’s time to do a patch, minor, or major version bump.</p> It further also provides ways to specify if a commit contains a breaking change, via a git footer.</p> It may be easiest to explain this by just showing you the spec.</p> The commit contains the following structural elements, to communicate intent to the consumers of your library:</p> fix:</strong> a commit of the type</em> fix</code> patches a bug in your codebase (this correlates with PATCH</code></a> in Semantic Versioning).</li> feat:</strong> a commit of the type</em> feat</code> introduces a new feature to the codebase (this correlates with MINOR</code></a> in Semantic Versioning).</li> BREAKING CHANGE:</strong> a commit that has a footer BREAKING CHANGE:</code>, or appends a !</code> after the type/scope, introduces a breaking API change (correlating with MAJOR</code></a> in Semantic Versioning). A BREAKING CHANGE can be part of commits of any type</em>.</li> types</em> other than fix:</code> and feat:</code> are allowed, for example @commitlint/config-conventional</a> (based on the Angular convention</a>) recommends build:</code>, chore:</code>, ci:</code>, docs:</code>, style:</code>, refactor:</code>, perf:</code>, test:</code>, and others.</li> footers</em> other than BREAKING CHANGE: <description></code> may be provided and follow a convention similar to git trailer format</a>.</li> </ol> Additional types are not mandated by the Conventional Commits specification, and have no implicit effect in Semantic Versioning (unless they include a BREAKING CHANGE).</p> A scope may be provided to a commit’s type, to provide additional contextual information and is contained within parenthesis, e.g., feat(parser): add ability to parse arrays</code>.</p> </blockquote> So basically, if there is a type fix</code>, that’s a PATCH</code> bump to your semver, a feat</code> is a MINOR</code> bump, and a BREAKING CHANGE:</code> footer or a !</code> after the type/scope is a MAJOR</code> bump.</p> If we for instance have commit with the summary fix: change windows-only imports to be windows-only</code>, that’s a PATCH</code> change, and if we are on version 3.2.48</code>, then next version will have to be at least 3.2.49</code>.</p> If we instead had a summary feat: add --no-\|show-symlinks flags for filtering output</code>, then that would a MINOR</code> change, and again, if we were on 3.2.48</code>, next release would be 3.3.0</code>.</p> Lastly, if we had a summary feat(flags)!: add --classify=always,auto,never</code>, even thou it has a feat</code> type, the !</code> indicates this is a breaking change, and so we must cut a new MAJOR</code> version on next release if we include this commit, meaning that if we were on 3.2.48</code>, we’d go to 4.0.0</code>.</p> Yes. Breaking changes means a new major version. Yes, that might seem sever to you if you’re not used to being at the helm of something being used by a lot of people, but that’s how we make these sausages (at least if we’re actually trying to do it correctly). The guy who authored the semver spec, Tom Preston-Werner</a> (also known for co-founding GitHub</a> and creating gravatar</a>) has a blog post about this called Major Version Numbers are Not Sacred</a>.</p> What people end up doing in practice is building up a lot of breaking changes and then releasing them all at once, to not constantly destroy the API/ABI and UX of their software. How this ties in with a more continuous development is a very interesting question, e.g. how does eza</a> manage to release a new version every week, yet also work on breaking changes like config files or moving to a new command line parser (specifically clap</a>). I think such a discussion is far out of scope for this post, but I may write about it in the future.</p> Conventional Commits and Linguistic Relativity</h2> If you were to ask wikipedia what “Behavioral Design” means, it would reply:</p> Behavioural design is a sub-category of design, which is concerned with how design can shape, or be used to influence human behaviour.</p> </blockquote> Now, how is that relevant to us as developers, maintainers, or just hobbyist hackers?</p> Well you see, the design of conventional commits, whether intentional or not, has a lot of decisions that end up shaping how contributors interact with your project.</p> When I initially forked eza</code>, I mostly moved us to conventional commits fairly early because I was concerned with a maintenance burden. I knew that exa</code> had been a project of maintainer burnout, and what I was trying to do was create longevity for the project. That meant that if there was something I could offload to someone or something else, I would, including changelogs.</p> And yes, conventional commits means I don’t have to write changelogs, I can automate it, and it means I don’t have to think too hard about what version is next, it’s automatically generated. In fact, I can release eza every week because a release is just running two scripts, doing a cargo publish</code>, and a nix-update eza</code> in the nixpkgs</a> repo and opening a PR.</p> While that release velocity is great, I found another, much more interesting perk of using conventional commits. Namely that conventional commits lead to better commits, better commit summaries, better PRs, and better structured commits.</p> For instance, if a project just allows a commit summary to be an arbitrary string, what you’ll often end up with is either something like made extensions work</code>, a single commit that encompasses vast refactorings of the codebase, introduces a feature, fixes 3 different bugs, and is one big monolithic and unrevievable mess.</p> On the other hand, it also saves you from the series of wip</code>, wip 2</code>, formatted code and added quantum cryptography</code>, made compile</code>. These are equally unreviewable, and completely destroy your git log.</p> Now, you may have the discipline of projects like nixpkgs</a> of asking contributors to rebase their commits, so that they each represent unique atomic changes. But even then, what conventional commits gives you is a language to express what these changes actually are.</p> Before I had conventional commits as something I could just write without looking at the spec, I would genuinly struggle putting words to what my changes did. It made it hard to create good PRs to projects.</p> Now, it’s much easier. I just go:</p> refactor(parser): used enums instead of bools in match logic</code></li> fix(parser): capital letters not differentiated</code></li> fix(parser): long lines ignored</code></li> feat(parser): add long flags</code></li> docs(parser): document long flag usage</code></li> style: change flag rendering</code></li> ci: check parser flags workflow</code></li> </ul> Not only does this make my PR much more structured, but by getting into that habit, I find that my work also becomes much more organized. Instead of just working on some hodgepodge mix of things at the same time and committing either to save my work or to put all of it together in a disorganized sack of a single commit, I find that I am much more able to dissect a problem into discrete solvable steps.</p> I’d even go as far as saying this makes you a better programmer, as breaking down a large problem into small solvable chunks is a great problem solving strategy.</p> What conventional commits does is it gives you this language to describe the steps it takes to solve a problem, and getting familiar with that vocabulary not only makes you a better programmer, but it makes it much easier to work with you, and it heavily reduced the burden on maintainers.</p> Linguistic relativity is the idea that the structure of a language influences the speakers world view and cognition, changing how you interact with your surroundings. Linguistic relativity is a hypothesis, but I find it pretty easy to prove the weak case that if you can’t signify something, you will struggle to operate with it.</p> In other words, conventional commits get you merged faster, and makes your code better. Hopefully you’re sold by this point.</p> There are some tools for dealing with these, such as git-footers</a>. Personally, I don’t really use those. But it’s good to know they exist. ↩</a></p> </li> </ol> rustls and aws-sdk seems incompatible with FOSS Christina Sørensen — Mon, 17 Jun 2024 14:00:00 +0000 I should preface this with the fact that I am by no means a lawyer or anything like it. Nothing here constitutes a legal argument or should be taken as true. I may be completely wrong or have missed crucial details.</p> As part of a rust project I’ve been working on at dayjob, I’ve had to audit all dependencies for license compatibility with the AGPLv3, our preferred license. Sadly, I encountered what I think is a major ecosystem blocker, and we will likely not be able to license the code under AGPLv3, or any GPL license for that matter, having to go with a more permissive, but compatible license until the problem this post is about is fixed. Which is a shame, and something I wanna bring more attention to.</p> A fairly fundamental crate in rust is ring</a>, providing “safe, fast, small crypto using Rust”. To understand how fundamental it is, consider that other crates such as rustls</a>, the aws-sdk-rust</a> crate, and sct</a> make use of ring. And this is by no means an exhaustive list.</p> The problem with ring, as you may well notice if you dare read its rather convoluted license</a>[1]</a></sup>, is that amongst it’s licenses is the OpenSSL license. The problem with the OpenSSL license is it’s equivalent to a BSD4 clause license. That is, it contains the problematic advertisement clause.</p> 3. All advertising materials mentioning features or use of this </span> * software must display the following acknowledgment: </span> * "This product includes software developed by the OpenSSL Project </span> * for use in the OpenSSL Toolkit. (http://www.openssl.org/)" </span></code></pre> This clause is well known</a> for being GPL-incompatible, and indeed, if you look up the OpenSSL license on the gnu.org</a> list of gpl compatible licenses</a>, it is indeed the case that the license is considered GPL-incompatible.</p> But so what? Big deal?</p> Big deal… yes. This means that if you want to interface with s3 storage[2]</a></sup>, or use rustls, you’ll be GPL incompatible!</p> For example, consider the following GNU Affero General Public License Version 3 projects that use crates in the aws-sdk</code> suite, or rustls</code>.</p> zed</a></li> Signal-Calling-Service</a></li> garage s3</a></li> quickwit</a></li> rustdesk</a></li> </ul> And these are just some quick examples I found with a quick github search, there is probably much more software currently out there in the rust ecosystem that’s using a license that its crates are fundamentally incompatible with[3]</a></sup>.</p> While this is a problem the ring</code> crate seems to be aware of</a>, it’s sad to see that there is such a large wound on the FOSS compatibility of the rust crate ecosystem. Not to put any blame anywhere. I am well aware that if this was easy to fix, it wouldn’t be a problem anymore, and anyone wanting to criticize the ring devs should spend the effort helping them fix the situation, rather than complaining, they seem eager to solve this. But it is something I think needs more awareness.</p> Update 2024-06-19: We’ll try to go with EUPL-1.2, as it is compatible with all BSD licenses. Also, it seems to have many other useful properties. This does not make the problem disappear however, but it does create a path forward for a stronger license for rust projects affected that want SaaS loophole protections.</p> This is a problem the ring devs are working on. Let it be a lesson to start your project off using REUSE</a> or paying the price down the line >:3 ↩</a></p> </li> Yes, there are other crates than the aws-sdk, but when investigating this for the project in question, none seem to be tennable solutions. Vet your dependencies! ↩</a></p> </li> Yes, there are potential ways to mitigate this, but by a quick skim, I haven’t seen any of those implemented in these projects, and implementing them would likely be both architecturally ugly, and horribly inefficient. For… reasons, I’ll not actually elaborate on how you could potentially circumvent licenses like this, because I’m really not a lawyer. ↩</a></p> </li> </ol> Merge or Rebase Christina Sørensen — Wed, 29 May 2024 13:40:00 +0000 This isn’t the article that’s gonna explain merge vs rebase for people that haven’t used git before (sorry). Rather, this is gonna be about some more meta things for maintainers, release engineers, and other people who care about development process.</p> Git gives you a lot of options. How you use it isn’t prescribed in advance. There are a lot of ways that people might use git differently, but here, I’m more seeking to investigate some of the ways of using git that are relevant to people in charge of git repositories with a non-trivial amount of contributors, like myself.</p> While I enjoy having these things done differently between projects I’m working in and part of maintaining — as it gives some diversity to the methods I get to use in my work, and let’s me continuously evaluate multiple methods[1]</a></sup> in practice — I do think it makes sense that for a single project, there is consistency in the way it’s developed, so that expectations can be set, and procedures can be learned and become second nature.</p> There are two main questions for a project to decide when it comes to merging via git.</p> How to merge feature branch changes from PRs.</li> How to merge newest changes into outdated branches.</li> </ol> I’ll discuss both, and then summariese my observations.</p> Merging branches</h2> When some developer starts working on a software feature, especially in open source, those changes, typicall, will be made in a separate branch that has branched off from the main branch. This is often called a feature branch. This allows the developer to work on their feature in isolation, and, at some point, put those changes back into the main branch, for all people to use.</p> This is what some may call trunk based development, although the lines get fuzzy about how long a PR should live. That discussion is out of scope for this post, but I’d love to follow up with a post about that at some point.</p> When moving changes from your feature branch into main, you can have potentially three possible options for merging.</p> Fast Forward</li> Rebase and Fast Forward</li> Merge</li> </ul> For fast forward, main may not have changed while you were working, and so you can “fast forward” your changes on top of main, without having to worry about any conflicts. This is, for both developers and maintainers, extremely easy, with some reservations that will be discussed later.</p> Another option is to take the changes from the updated main branch, and git pull --rebase origin main</code> them into your feature branch, placing your changes on top of the latest changes from main. As you may guess, this allows you to git merge --ff-only</code> your changes into the main branch.</p> The final strategy is the classical merge. You may decide to merge your changes, and the changes in main together, using a special commit that represents the operation of merging the two together, a merge commit. Such a commit has special properties, such as having two parents, and being the result of the unification of two separate branches.</p> Of course, if you’re not doing just a simple fast forward, either a rebase-ff or merge may require that you solve potential code conflicts. For instance, someone may have changed a line that you have also changed, leading to the two versions colliding. For the merge, you resolve this by making the merge commit define the merged version, for a rebase, you’ll go through your changes and adjust them so they fit cleanly onto the branch you’re pulling.</p> One important difference to note for us, is between the two, merging into main typically gets done by the maintainer, where as rebasing gets done by the developer working on the feature branch.</p> Rebasing also has some downsides. For example, if the person rebasing doesn’t have the PGP private key of the people who’s commit are included in the rebase, they cannot sign the commits. This means that if Alice made 3 commits and signed them with her PGP signature, Bob can’t later rebase those changes onto some other commit without breaking that signature. This is because, while the code doesn’t change, the commit revision does, and so the signature is wrong.</p> Merging does not have this issue, as a merge is simply the actual merge commit, containing the new state of the branch, that potentially specifies how to rearrange everything so as to not cause any conflicts. Thus it doesn’t suffer from breaking the signatures.</p> Now, if you’re using rebase for merges into main, as we do in eza, you may still want a signature verifying that a trusted person has indeed verified these changes are up to whatever standard they guarantee (eza at best guarantees that it runs on my linux box that you should be able to install it on NixOS[2]</a></sup>). The way we solve this is by tagging all our releases with a signed tag, making sure you know that I’ve personally said that this is the codebase I want you to run for a given release.</p> </p> But even then, it’s coping. It’s not as nice as having a tree full of the signatures of everyone that’s committed, right on their commits.</p> Another point raised against rebases is they rewrite history. This is true, unlike a merge that just merges two branches and keeps both in history, the rebase will make it appear as thou the feature branch was developed on top of the main branch. I don’t personally think this is a huge concern, except the loss of potential pretty git log --graph</code> outputs.</p> One of the annoyances of merges, in comparison, is that they’re very noisy. Consider this output from nixpkgs:</p> git</span></span> log --</span>oneline</span></span> </span>bfb7a882678e</span></span> Merge pull request #</span></span>314109 from trofi/githooks.tests-fix-eval</span> </span></span></span>47bdef656cc5</span></span> Merge pull request #</span></span>311260 from purepani/update-svelte-language-tools</span> </span></span></span>fc165a03b23a</span></span> Merge pull request #</span></span>309524 from r-ryantm/auto-update/enet</span> </span></span></span>3de810d52cbe</span></span> Merge pull request #</span></span>309724 from r-ryantm/auto-update/secp256k1</span> </span></span></span>1d6a2f5a4d4d</span></span> Merge pull request #</span></span>309787 from r-ryantm/auto-update/python311Packages.minio</span> </span></span></span>a36fa5451c44</span></span> Merge pull request #</span></span>309426 from Rconybea/add-sphinxcontrib-ditaa</span> </span></span></span>890f8e436a56</span></span> Merge pull request #</span></span>309476 from r-ryantm/auto-update/goperf</span> </span></span></span>dfa36c1d67f5</span></span> Merge pull request #</span></span>312557 from r-ryantm/auto-update/intune-portal</span> </span></span></span>7d8ed5ce921d</span></span> Merge pull request #</span></span>291853 from greaka/grafana</span> </span></span></span>d0a20d7c5955</span></span> Merge pull request #</span></span>314112 from khaneliman/bicep</span> </span></span></span>4d2462511f06</span></span> Merge pull request #</span></span>314099 from mrkline/snapper-and-borgbackup-doc-fix</span> </span></span></span>3e3ac0e7baa8</span></span> Merge pull request #</span></span>305516 from OPNA2608/init/lomiri/ayatana-indicator-display</span> </span></span></span>a46ce7c77d9d</span></span> svelte-language-server: convert to buildNpmPackage</span> </span>7962cbb2326b</span></span> Merge pull request #</span></span>314082 from superherointj/bicep-0.27.1</span> </span></span></span>5771dbfa7d2f</span></span> bicep: fix updater script</span> </span>2db4e7d035f8</span></span> Merge pull request #</span></span>313729 from pluiedev/zhf-24.05/gobang</span> </span></span></span>44744fc83f3c</span></span> githooks.tests: fix eval</span> </span>0f3add331c6a</span></span> segger-jlink: 794l -></span> 796</span>b</span> </span></code></pre> This… isn’t nice, and makes history way harder to read. Compare to eza:</p> git</span></span> log --</span>oneline</span></span> </span>22dbeec2</span></span> (HEAD -></span> main, origin/main, origin/HEAD</span></span>) build</span></span>(deps</span></span>):</span></span> bump libc from 0.2.154 to 0.2.155</span> </span>82e4f0f3</span></span> build(deps</span></span>):</span></span> bump trycmd from 0.15.1 to 0.15.2</span> </span>bbede6d4</span></span> (tag: v0.18.16</span></span>) chore:</span></span> release eza v0.18.16</span> </span>70891fa1</span></span> docs(README</span></span>):</span></span> use 3 columns for packaging status badge</span> </span>e8e0b6da</span></span> fix: change windows-only imports to be windows-only</span> </span>15528d8b</span></span> docs(install</span></span>):</span></span> fix typo in `</span>INSTALL.md</span></span>`</span></span></span> </span>26e5943a</span></span> docs: update INSTALL.md</span> </span>20241984</span></span> build(deps</span></span>):</span></span> bump DeterminateSystems/nix-installer-action from 10 to 11</span> </span>feff970d</span></span> docs(man</span></span>):</span></span> replace decay with color-scale</span> </span>a1c36389</span></span> build(deps</span></span>):</span></span> bump DeterminateSystems/flake-checker-action from 5 to 7</span> </span>7437c2a1</span></span> (tag: v0.18.15</span></span>) chore:</span></span> release eza v0.18.15</span> </span>f1ef455b</span></span> feat(devtools</span></span>):</span></span> add optional tag argument to deb-package.sh</span> </span>23502d3c</span></span> fix(devtools</span></span>):</span></span> correct command for latest tag in deb-package.sh</span> </span>292cf7ec</span></span> feat(devtools</span></span>):</span></span> return to original commit at the end of deb-package.sh</span> </span>4f542eaf</span></span> docs(reaedme</span></span>):</span></span> add some keywords for benefit of ctrl-f</span> </span>6d6612d2</span></span> docs(readme</span></span>):</span></span> move heading out of collapsed section</span> </span>abe9f587</span></span> docs(readme</span></span>):</span></span> correct heading levels in markdown</span> </span>01a9c2aa</span></span> docs(readme</span></span>):</span></span> add how to find man pages in terminal and online. Partly fixes #</span></span>967</span> </span></span></span>f78e4bb6</span></span> (tag: v0.18.14</span></span>) chore:</span></span> release eza v0.18.14</span> </span>01919cdd</span></span> build(deps</span></span>):</span></span> bump palette from 0.7.5 to 0.7.6</span> </span>414f70ba</span></span> build(deps</span></span>):</span></span> bump unicode-width from 0.1.11 to 0.1.12</span> </span>1554472a</span></span> build(deps</span></span>):</span></span> bump libc from 0.2.153 to 0.2.154</span> </span>bebd39c0</span></span> build(deps</span></span>):</span></span> bump uzers from 0.11.3 to 0.12.0</span> </span>df7d51fc</span></span> feat: add icon for "</span>cron.minutely"</span></span> directory</span> </span>8b7dc5f5</span></span> (tag: v0.18.13</span></span>) chore:</span></span> release eza v0.18.13</span> </span>c0df8ecd</span></span> feat: generate completion/manpage tarballs on release</span> </span>8afb5cc2</span></span> (tag: v0.18.12</span></span>) fix:</span></span> checking for deref flag in file_name</span> </span>a4782d1d</span></span> feat: add scheme filetype and icons</span> </span>87b36785</span></span> fix: allow unused imports for freebsd</span> </span>99562e3a</span></span> (tag: v0.18.11</span></span>) chore:</span></span> release eza v0.18.11</span> </span>07f67708</span></span> fix: build aarch64, arm without libgit2</span> </span>17733e9a</span></span> fix(netbsd</span></span>):</span></span> enable the rule only for NetBSD.</span> </span>7664a1fb</span></span> ci: bump NetBSD version to 10.0</span> </span>462fc344</span></span> fix(netbsd</span></span>):</span></span> fix clippy lints</span> </span>75f1f8cf</span></span> (tag: v0.18.10</span></span>) chore:</span></span> release eza v0.18.10</span> </span></code></pre> Isn’t that a lot nicer? I think so.</p> Of course, what may be lost is that it’s no longer obvious from what PR a change originated. It also isn’t obvious to me when some feature branch was created, how long it was worked on and so on. This isn’t… really that important either, but it’s nice to have in logs. But you can just look at branches if you need to know this for tooling or something.</p> For nixpkgs, you get a to have a visual overview of how branches end up in main. It looks like this:</p> git</span></span> log --</span>oneline</span> --</span>graph</span></span> </span></span></span> 4d4571b20a29 (HEAD -></span> devpi-loadcredential</span></span>) nixos/devpi-server:</span></span> fix loading credentials as DynamicUser</span> </span></span></span> bfb7a882678e Merge pull request #</span></span>314109 from trofi/githooks.tests-fix-eval</span> </span></span></span></span>\|</span>\ </span> </span></span>\|</span> </span></span> 44744fc83f3c githooks.tests: fix eval</span> </span></span></span> \|</span> 47bdef656cc5</span></span> Merge pull request #</span></span>311260 from purepani/update-svelte-language-tools</span> </span></span></span></span>\|</span>\ </span>\ </span> </span></span>\|</span> </span></span> \|</span> a46ce7c77d9d</span></span> svelte-language-server: convert to buildNpmPackage</span> </span></span></span> \|</span> </span>\|</span> fc165a03b23a</span></span> Merge pull request #</span></span>309524 from r-ryantm/auto-update/enet</span> </span></span></span></span>\|</span>\ </span>\ </span>\ </span> </span></span>\|</span> </span></span> \|</span> </span>\|</span> 8e729c70a119</span></span> enet: 1.3.17 -></span> 1</span>.3.18</span> </span></span></span> \|</span> </span>\|</span> </span>\|</span> 3de810d52cbe</span></span> Merge pull request #</span></span>309724 from r-ryantm/auto-update/secp256k1</span> </span></span></span></span>\|</span>\ </span>\ </span>\ </span>\ </span> </span></span>\|</span> </span></span> \|</span> </span>\|</span> </span>\|</span> f96a90834c5e</span></span> secp256k1: 0.4.1 -></span> 0</span>.5.0</span> </span></span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> 1d6a2f5a4d4d</span></span> Merge pull request #</span></span>309787 from r-ryantm/auto-update/python311Packages.minio</span> </span></span></span></span>\|</span>\ </span>\ </span>\ </span>\ </span>\ </span> </span></span>\|</span> </span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> 24d453186792</span></span> python311Packages.minio: 7.2.6 -></span> 7</span>.2.7</span> </span></span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> a36fa5451c44</span></span> Merge pull request #</span></span>309426 from Rconybea/add-sphinxcontrib-ditaa</span> </span></span></span></span>\|</span>\ </span>\ </span>\ </span>\ </span>\ </span>\ </span> </span></span>\|</span> </span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> 197d6261ebf8</span></span> python311Packages.sphinxcontrib-ditaa: init at 1.0.2</span> </span></span>\|</span> </span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> 51de558f5634</span></span> maintainers: add rconybea</span> </span></span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> 890f8e436a56</span></span> Merge pull request #</span></span>309476 from r-ryantm/auto-update/goperf</span> </span></span></span></span>\|</span>\ </span>\ </span>\ </span>\ </span>\ </span>\ </span>\ </span> </span></span>\|</span> </span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> 8eea66b3f69a</span></span> goperf: 0-unstable-2023-11-08 -></span> 0</span>-unstable-2024-05-10</span> </span></span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> dfa36c1d67f5</span></span> Merge pull request #</span></span>312557 from r-ryantm/auto-update/intune-portal</span> </span></span></span></span>\|</span>\ </span>\ </span>\ </span>\ </span>\ </span>\ </span>\ </span>\ </span> </span></span>\|</span> </span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> 010c4a334eaf</span></span> intune-portal: 1.2404.23-jammy -></span> 1</span>.2404.25-jammy</span> </span></span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> 7d8ed5ce921d</span></span> Merge pull request #</span></span>291853 from greaka/grafana</span> </span></span></span></span>\|</span>\ </span>\ </span>\ </span>\ </span>\ </span>\ </span>\ </span>\ </span>\ </span> </span></span>\|</span> </span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> 254dbdcc6296</span></span> grafanaPlugins.grafana-oncall-app: init at 1.5.1</span> </span></span>\|</span> </span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> 0e5f44658ee6</span></span> maintainers/team-list: add fslabs</span> </span></span>\|</span> </span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> 8d6f8c9ed75a</span></span> maintainers: add lpostula</span> </span></span>\|</span> </span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> 7bda925dacb2</span></span> maintainers: add greaka</span> </span></span></span> \|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> </span>\|</span> d0a20d7c5955</span></span> Merge pull request #</span></span>314112 from khaneliman/bicep</span> </span></span></span></code></pre> Another problem worth considering is that a rebase may — as Astrid</a> pointed out — results in main being in a state where a git bisect</code> of a build command may fail, if you haven’t ensured that all commits in the feature branch still worked after being rebased on top of the latest main.</p> Now this is rare, but rare isn’t really a good argument for not caring here, as breaking once would be extremely annoying for people working on a bugfix, leading to false git dissect</code> positives.</p> There are solutions to this. You could (perhaps) have each commit in a branch run against CI. This may be outside of your compute budget thou. Another option is to run your build command, e.g. cargo build</code> on each commit after a rebase, but then, this kind of checking step usually falls on the maintainer, and we, as maintainers, should seek to limit our own burdens as much as possible (to prevent burnout[3]</a></sup>, the killer of most FOSS projects).</p> To eza, this isn’t fatal, but a very interesting point worth keeping in mind.</p> Updating a branch</h2> Of course, since there may be multiple people all working on separate branches, there is another problem. One person might have worked on something, at the same time as you worked on something else, and then gotten their changes merged before you had a chance. But if that happens, your changes might now conflict with what was in the main branch when you started, and worse, their changes may make ruin your work, or even make it unable to compile!</p> The first choice a project should make, is whether or not to allow outdated feature branches to be merged. The problem with tolerating outdated branches is you will not be able to fix any problems related to integrating the code with the main codebase before merging it. This puts the burden of ensuring the code works on the maintainer, rather than the developer, and this is bad!</p> If you do choose to not allow this, another question faces you: how should the main branch be merged into the feature branch when the feature branch is out of date with the main branch.</p> Your options are either to merge the main branch into your feature branch, or to rebase your changes on top of main, making it like if you had started working on the latest changes.</p> To me, it’s a more simple dilemma, merges shouldn’t exist if they don’t meaningfully represent merging two things together. Simply updating your branch is what rebase does best, and having merges both from feature branches into main and from main into your feature branch is absolutely tasteless. This clutters your history for no reason. It also makes it harder for you to fix your commits when the reviewer tells you that you had a faulty commit 3 commits before you merged main into your feature branch and kept working for 4 more commits.</p> In general, rebasing with merge commits is tedious, and we only consider merges viable for the main branch under the premise that we won’t later need to rebase main, because everything that gets into main is in a state of being up to standard.</p> Sadly, code forges lack a feature to only allow updating branches with the projects preferred standard, unlike for merges where rebase, squash, and merge are all options, that all can be disabled by the repositories owner(s).</p> Summary</h2> So what where the tradeoffs?</p> Merging feature branches into main Merges. Preserves history.</li> Gives you git log --graph</code>.</li> Doesn’t break PGP signatures.</li> Creates ugly git log</code>.</li> Gives weak protection that all commits build (if you have CI)</li> Puts burden of verification on maintainer.</li> </ul> </li> Rebase. Rewrites history.</li> No git log --graph</code>.</li> Breaks PGP signatures.</li> Creates useful git log</code> output.</li> Needs more care if you want all commits to be validated.</li> Puts burden of verification on committer (mostly).</li> </ul> </li> Fast Forward. Like a rebase, but with less drawback.</li> </ul> </li> </ul> </li> Updating your feature branch. Merges. merges make git rebase -i</code> difficult, when asked to change your commits.</li> It makes your git log</code> unreadable.</li> Doesn’t provide any benefits.</li> Mostly just a quirk that git even allows this, it doesn’t make sense.</li> </ul> </li> Rebase. The preffered choice.</li> Fits like a glove for updating your feature branch.</li> </ul> </li> </ul> </li> </ul> As a maintainer, remember that what where really trying to do if offload any non-maintainer burdens onto relavant roles, like developers and reviewers. We may wear multiple hats, often we are the final reviewers, but even then, it shouldn’t be our job to ensure a clean merge, we should meerly verify it.</p> The main goal is always: maintainers shouldn’t need to fix the feature branch!</p> This also applies for formatters. I personally mostly use alejandra for nix, but dayjob uses nixfmt (the RFC 166 version). I just hit a format button before commiting anyways, so it’s just cool to see the different options and feel out which ones I truly prefer. ↩</a></p> </li> Because I don’t have any MacOS, Windows, or BSD boxes to test on, and we don’t have any maintainers that do with the spoons to be involved in giving such guarantees. ↩</a></p> </li> Consider reading this excellent resource https://opensource.guide/maintaining-balance-for-open-source-maintainers/</a>, and this mozilla worksheet https://docs.google.com/document/d/1esQQBJXQi1x_-1AcRVPiCRAEQYO4Qlvali0ylCvKa_s/edit?pli=1#heading=h.xd8n2v3f4866</a>. ↩</a></p> </li> </ol> Grok Git Repos Christina Sørensen — Wed, 15 May 2024 00:00:00 +0000 Introduction</h1> Git is often perceived as a necessary evil, a tool to be tolerated rather than mastered.</p> But treating your primary work tool as a mystery is a vulnerability. When the abstraction leaks — and it always does — the developer who understands the internals can surgically repair the damage. In contrast, those who rely on surface-level knowledge often leave behind a sloppy timeline. A history riddled with vague messages and botched merges signals a lack of care that code quality alone cannot hide.</p> Unfortunately, we often learn git through folklore — memorizing obscure commands to save us when things go wrong, without understanding why they work. We rely on luck rather than logic.</p> To escape this ritualistic dependency, we must strip away the interface. To truly grok</em> git, we will ignore the high-level commands and build a repository from the ground up, proving that the magic is actually just simple, elegant engineering.</p> Creating a Git Repository from Scratch</h1> Let's create a folder for us to work in.</p> $</span></span> mkdir grok</span> </span>$</span></span> cd grok</span> </span></code></pre> Now, we have a folder. A folder isn't a git repository, as we can verify.</p> $</span></span> git status</span> </span>fatal:</span></span> not a git repository (or any of the parent directories</span></span>):</span></span> .git</span> </span></code></pre> As the status</code> subcommand tells us, neither is the directory we currently created, of any of it parent directories a git repository.</p> If that wasn't the case for you, that would be because you created the folder grok</code> as a subdirectory of a git repository.</p> </blockquote> The error seems to mention a .git</code> folder. Turns out, when speaking of a git repository, we are actually referring to this folder in most cases[1]</a></sup>. So, in theory, it should be simple enough to create a git repository right, just create a .git</code> folder and we are good to go?</p> $</span></span> mkdir .git</span> </span>$</span></span> git status</span> </span>fatal:</span></span> not a git repository (or any of the parent directories</span></span>):</span></span> .git</span> </span></code></pre> Ohh, apparently not so.</p> If we were to look up the definition of a repository</em> in the git glossary</a>, we would see the following.</p> A collection of refs together with an object database containing all objects which are reachable from the refs, possibly accompanied by meta data from one or more porcelains. A repository can share an object database with other repositories via alternates mechanism.</p> </blockquote> Ahh, simple. Seems like we need to have some refs and an object database with all the objects refs can reach. Also, there might be meta data from porcelains?</p> Although the definition may seem complex, the underlying concept is straightforward. Rather than summarizing it briefly, we will demonstrate its simplicity through a practical, step-by-step example.</p> First, we need an object database</em>. Well, database sounds a bit hard, we could just make a folder for it instead in our git repository.</p> $</span></span> mkdir .git/objects</span> </span></code></pre> Now, we also need a collection of refs</em>. No issue, just make a folder for them.</p> $</span></span> mkdir -</span>p</span> .git/refs/heads</span> </span></code></pre> What are heads? Don't worry, we'll get to it. Now we have the bare minimum for a repository thou, right?</p> Well, let's try to find out!</p> $</span></span> git status</span> </span>fatal:</span></span> not a git repository (or any of the parent directories</span></span>):</span></span> .git</span> </span></code></pre> It appears we are missing a critical component.</p> $</span></span> echo "</span>ref: refs/"</span></span> ></span> .git/HEAD</span> </span></code></pre> Now, is this finally a repository? Let's see what git thinks the status is.</p> $</span></span> git status</span> </span>On</span></span> branch main</span> </span> </span>No</span></span> commits yet</span> </span> </span>nothing</span></span> to commit (create/copy files and use "</span>git add"</span></span> to track</span></span>) </span></code></pre> Seems like it is! …So now we have a git repository, that was easy</em>.</p> Committing to a Repository</h1> Creating Objects</h2> While creating a git repository manually is an interesting exercise, it is of little value unless we can populate it with files.</p> We likely will want to fill this repository with text files. For the occasion, you can use fortune to generate something, but if you use what I got, you'll be able to compare your hashes to the ones in the article.</p> "If a listener nods his head when you're explaining your program, wake him up.</p> </blockquote> That is very wise indeed. We want to add this to the repository. But how?</p> Well, one way to add it is the following, using the git hash-object</code> subcommand.</p> $ echo "If a listener nods his head when you're explaining your program, wake him up." \| git hash-object --stdin -w </span>665e95f1674e9466cb429bdfebaf1b8792ef0eec </span></code></pre> Okay, two questions now. What is 665e95f1674e9466cb429bdfebaf1b8792ef0eec</code>, and what did hash-object</code> just do?</p> We can inspect the git repository to investigate. Examining the directory tree structure provides some insight.</p> $</span></span> tree .git</span> </span>.git/</span></span> </span>├──</span></span> HEAD</span> </span>├──</span></span> objects</span> </span>│</span></span> └── 66</span> </span>│</span></span> └── 5e95f1674e9466cb429bdfebaf1b8792ef0eec</span> </span>└──</span></span> refs</span> </span> └──</span></span> heads</span> </span> </span>4</span></span> directories, 2 files</span> </span></code></pre> Here, we see that our object “database” in the objetcs folder has seen some change. There is a folder, 66</code>, with a file that has an equally vexing name, 5e95f1674e9466cb429bdfebaf1b8792ef0eec</code>.</p> I wonder what is in this file. Let's try to take a look at it.</p> $</span></span> cat .git/objects/66/5e95f1674e9466cb429bdfebaf1b8792ef0eec</span> </span>WKX</span></span> </span> Y$</span>b</span></span></span></span> `</span>݆!rB-8</span></span></span>}ɢ,xSOŋE598</span></span></span>}y?</span></span> </span></span></span> </span></span></span> %</span></span></span></span> </span></span></span></code></pre> The file does not contain plain text; it appears to be in a binary format. We can use the file</code> command to determine its type.</p> $</span></span> file .git/objects/66/5e95f1674e9466cb429bdfebaf1b8792ef0eec</span> </span>.git/objects/66/5e95f1674e9466cb429bdfebaf1b8792ef0eec:</span></span> zlib compressed data</span> </span></code></pre> Apparently, it's some data compressed with zlib. That means it's likely DEFLATE compressed (RFC 1951) in a zlib wrapper (RFC 1950)^{[2]</a></sup>.</p>} There is actually a neat hack we can use to view this data[3]</a></sup>.</p> $</span></span> printf "</span>\x1f\x8b\x08\x00\x00\x00\x00\x00"</span></span></span> \|</span> cat</span></span> - .git/objects/66/5e95f1674e9466cb429bdfebaf1b8792ef0eec</span> \|</span> gzip</span></span> -</span>dc</span></span> </span>blob</span></span> 78If a listener nods his head when you'</span>re explaining your program, wake him up. </span></span></span> </span></span></span>gzip: stdin: unexpected end of file </span></span></span></code></pre> We have located our string. The output begins with blob</code>, which specifies the git object’s file type, followed by 78</code>, indicating the blob’s size in bytes[4]</a></sup>.</p> Objects in git have 3 primary types:</p> blob</li> tree</li> commit</li> </ul> When we ran the hash-object</code> subcommand, we created a blob. Regarding the name, that is just the sha1 hash (RFC 3174) of the object, and some other metadata and such[5]</a></sup>. The actual output of the command was this hash. It is 40 characters long.</p> The name of the directory in the object database is the first two characters of the hash, that is 66</code>, and the actual object files name is the 38 other characters.</p> Manually decompressing objects for inspection is inefficient. Instead, we can use the hash 665e95f1674e9466cb429bdfebaf1b8792ef0eec</code> to inspect the newly created object.</p> Inspecting Objects</h2> We do this with the git cat-file</code> subcommand. To see the file type of an object, we use the type flag -t</code>.</p> $</span></span> git cat-file -</span>t</span> 665e95f1674e9466cb429bdfebaf1b8792ef0eec</span> </span>blob</span></span> </span></code></pre> That means that the type (-t</code>) of the object we made is a blob, as we saw by inspecting it. What if we wanna see the contents of the blob? We use -p</code> for print.</p> $</span></span> git cat-file -</span>p</span> 665e95f1674e9466cb429bdfebaf1b8792ef0eec</span> </span> </span>If</span></span> a listener nods his head when you'</span>re explaining your program, wake him up. </span></span></span> </span></span></span></code></pre> This procedure is significantly more convenient than the manual decompression hack used earlier.</p> Also, another useful flag to know is -s</code>, for size.</p> $</span></span> git cat-file -</span>s</span> 665e95f1674e9466cb429bdfebaf1b8792ef0eec</span> </span>78</span></span> </span></code></pre> This matches the 78</code> observed earlier, confirming the object size in bytes.</p> Index</h2> Okay, enough about the blob. It seems like we have added an object – with some text – into the git repository now. We might wonder if this is reflected in the git status</code> of the repo.</p> $</span></span> git status</span> </span>On</span></span> branch main</span> </span> </span>No</span></span> commits yet</span> </span> </span>nothing</span></span> to commit (create/copy files and use "</span>git add"</span></span> to track</span></span>) </span></code></pre> Adding an object to the database stores the data, but it does not tell git where that file belongs in your project. This is the purpose of the index</strong> (or staging area): to build the tree structure for the next commit. Without being registered in the index, our blob is essentially a “loose” object. If it remains unreferenced, it will eventually be removed by git gc</code> (garbage collection).</p> To actually commit our object, we must first register it in the index.</p> git</span></span> update-index --</span>add</span> --</span>cacheinfo</span> 100633 665e95f1674e9466cb429bdfebaf1b8792ef0eec truth.txt</span> </span></code></pre> The long string is the hash of the blob object we created, and truth.txt</code> is a fitting filename.</p> What about 100633</code>? Looking at man git update-index</code> we see that…</p> --cacheinfo <mode>,<object>,<path>, </span>--cacheinfo <mode> <object> <path> </span> </span>Directly insert the specified info into the index. </span>For backward compatibility, you can also give these </span>three arguments as three separate parameters, but </span>new users are encouraged to use a single-parameter </span>form. </span></code></pre> Although the documentation suggests the single-parameter form for new users, we have used the separate parameter form here for clarity. The value 100633</code> is the mode of the file. In git, this mode is a 6-digit octal number that defines the file type and permissions, similar to standard UNIX file modes but restricted to a few specific values.</p> But crucially, for blobs in git, we only have 3 modes available^{[6]</a></sup>:</p>} 100644 a normal file</li> 100755 a executable file</li> 120000 a symbolic link</li> </ul> Let's take a look at the repo structure again.</p> $</span></span> tree .git</span> </span>.git/</span></span> </span>├──</span></span> HEAD</span> </span>├──</span></span> index</span> </span>├──</span></span> objects</span> </span>│</span></span> └── 66</span> </span>│</span></span> └── 5e95f1674e9466cb429bdfebaf1b8792ef0eec</span> </span>└──</span></span> refs</span> </span> └──</span></span> heads</span> </span> </span>4</span></span> directories, 3 files</span> </span></code></pre> Here we notice that there now is an index</code> file. So all we did was add some index file</code>? …well, slow down. Let's do a sanity check with git status</code>.</p> $</span></span> git status</span> </span>On</span></span> branch main</span> </span> </span>No</span></span> commits yet</span> </span> </span>Changes</span></span> to be committed:</span> </span> (</span>use</span></span> "</span>git rm --cached <file>..."</span></span> to unstage</span>)</span> </span> new</span></span> file: truth.txt</span> </span> </span>Changes</span></span> not staged for commit:</span> </span> (</span>use</span></span> "</span>git add/rm <file>..."</span></span> to update what will be committed</span>)</span> </span> (</span>use</span></span> "</span>git restore <file>..."</span></span> to discard changes in working directory</span>)</span> </span> deleted:</span></span> truth.txt</span> </span></code></pre> Aha!</p> So we did something, we staged (added) the truth.txt file… but we also have unstaged changes saying we deleted it?</p> Well, yes. We have told git that the blob object we created is in the working-tree</em>, and it has the actual blob for this in its object database… but apparently can't find the file we told it about. This makes sense, since we just created the blob from stdin and didn't actually add any files.</p> Because the file exists in the index but is missing from the actual working directory, git correctly assumes it has been removed.</p> Hmm, while we are at it, let's take a look at .git/index</code>, just to see what that is about.</p> $</span></span> cat .git/index</span> </span>DIRCf^gNfB</span></span> truth.txtqblr33il%</span></span></span> </span></code></pre> This confirms it is not a text file. We can verify its type using the file</code> command.</p> $</span></span> file .git/index</span> </span>.git/index:</span></span> Git index, version 2, 1 entries</span> </span></code></pre> The output identifies it as a Git index, version 2</code> with 1 entries</code>. This single entry corresponds to the file we just added. To see what this actually looks like on disk, we can inspect the raw binary data.</p> $</span></span> hexdump -</span>C</span> .git/index</span> </span>44</span></span> 49 52 43 00 00 00 02 00 00 00 01 00 00 00 00</span> \|</span>DIRC............</span></span>\|</span> </span>00</span></span> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00</span> \|</span>................</span></span>\|</span> </span>00</span></span> 00 00 00 00 00 81 a4 00 00 00 00 00 00 00 00</span> \|</span>................</span></span>\|</span> </span>00</span></span> 00 00 00 66 5e 95 f1 67 4e 94 66 cb 42 9b df</span> \|</span>....f^..gN.f.B..</span></span>\|</span> </span>eb</span></span> af 1b 87 92 ef 0e ec 00 09 74 72 75 74 68 2e</span> \|</span>..........truth.</span></span>\|</span> </span>74</span></span> 78 74 00 c1 b9 71 62 b4 ab c1 c2 6c 72 33 33</span> \|</span>txt...qb....lr33</span></span>\|</span> </span>92</span></span> 69 92 99 ac 6c 8e ff</span> \|</span>.i...l..</span></span>\|</span> </span> </span></code></pre> (I removed the offset from the output)</p> The first 4 bytes, DIRC</code>, are the “magic number” — a constant value used to identify the file format (short for “Directory Cache”). Following some padding bytes[7]</a></sup>, the 02</code> indicates the version number. The rest of the file contains the entry data, including the filename truth.txt</code>.</p> If we were to add another file, likely patterns would emerge in the binary structure. Investigating how the index looks in a larger project is an excellent way to learn more, so I have included it in the Exercises</em> section below.</p> Tree Objects</h2> Back on track, we've now made the index thingy, and we'll have to write it with the write-tree</code> subcommand.</p> $</span></span> git write-tree</span> </span>a6325f064bac723691f20c0b1ed2bea82a1728fd</span></span> </span></code></pre> git status</code> does not seem to have changed, but if we check what's in repo's file structure, we'll see something did change.</p> $</span></span> tree .git</span> </span>.git</span></span> </span>├──</span></span> HEAD</span> </span>├──</span></span> index</span> </span>├──</span></span> objects</span> </span>│</span></span> ├── 66</span> </span>│</span></span> │ └── 5e95f1674e9466cb429bdfebaf1b8792ef0eec</span> </span>│</span></span> └── a6</span> </span>│</span></span> └── 325f064bac723691f20c0b1ed2bea82a1728fd</span> </span>└──</span></span> refs</span> </span> └──</span></span> heads</span> </span> </span>5</span></span> directories, 4 files</span> </span></code></pre> We notice the a6325f064bac723691f20c0b1ed2bea82a1728fd</code> sha-1 hash refers to a new object. Interesting, wonder what this is.</p> To thoroughly understand this object, we will inspect it manually by examining the raw file. This is the “hard way,” but it offers the most insight into the underlying data structure.</p> $</span></span> file .git/objects/a6/325f064bac723691f20c0b1ed2bea82a1728fd</span> </span>.git/objects/a6/325f064bac723691f20c0b1ed2bea82a1728fd:</span></span> zlib compressed data</span> </span></code></pre> Okay, that's zlib compressed data. We know that, so let's use that same hack to figure out what is inside of this blob.</p> $</span></span> printf "</span>\x1f\x8b\x08\x00\x00\x00\x00\x00"</span></span></span> \|</span> cat</span></span> - .git/objects/a6/325f064bac723691f20c0b1ed2bea82a1728fd</span> \|</span> gzip</span></span> -</span>dc</span></span> </span>tree</span></span> 37100644 truth.txtf^gNfB</span> </span>gzip:</span></span> stdin: unexpected end of file</span> </span></code></pre> This object is not a blob, but a tree</em> — one of the three primary git object types. The number 37100644</code> initially appears confusing, but it is actually two separate values concatenated: the size (37</code> bytes) and the file mode (100644</code>). Following this are the filename and a binary sequence f^gNfB</code>.</p> The sequence f^gNfB</code> matches data we previously observed in the hexdump</code> of the .git/index</code> file. Identifying its specific purpose is left as an exercise for the reader.</p> Let's look at the contents the canonical way.</p> $</span></span> git cat-file -</span>p</span> a6325f064bac723691f20c0b1ed2bea82a1728fd</span> </span>100644</span></span> blob 665e95f1674e9466cb429bdfebaf1b8792ef0eec truth.txt</span> </span></code></pre> An Experiment with Tree Objects</h2> An important concept to explore is how a tree object handles multiple files. To demonstrate this, we will create another blob.</p> $</span></span> echo "</span>AMOGUS"</span></span></span> \|</span> git</span></span> hash-object --</span>stdin</span> -</span>w</span></span> </span>f58617716d903fb842b5606a335ff1406b9a21d3</span></span> </span></code></pre> And add it to our index file.</p> $</span></span> git update-index --</span>add</span> --</span>cacheinfo</span> 100633 f58617716d903fb842b5606a335ff1406b9a21d3 amogus.txt</span> </span> </span></code></pre> Now, let's look at the repository. We have 3 objects in our objects database.</p> $</span></span> tree .git</span> </span>.git</span></span> </span>├──</span></span> HEAD</span> </span>├──</span></span> index</span> </span>├──</span></span> objects</span> </span>│</span></span> ├── 66</span> </span>│</span></span> │ └── 5e95f1674e9466cb429bdfebaf1b8792ef0eec</span> </span>│</span></span> ├── a6</span> </span>│</span></span> │ └── 325f064bac723691f20c0b1ed2bea82a1728fd</span> </span>│</span></span> └── f5</span> </span>│</span></span> └── 8617716d903fb842b5606a335ff1406b9a21d3</span> </span>└──</span></span> refs</span> </span> └──</span></span> heads</span> </span> </span>6</span></span> directories, 5 files</span> </span></code></pre> Let's dump the index binary.</p> </span>$</span></span> hexdump -</span>C</span> .git/index</span> </span>44</span></span> 49 52 43 00 00 00 02 00 00 00 02 00 00 00 00</span> \|</span>DIRC............</span></span>\|</span> </span>00</span></span> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00</span> \|</span>................</span></span>\|</span> </span>00</span></span> 00 00 00 00 00 81 a4 00 00 00 00 00 00 00 00</span> \|</span>................</span></span>\|</span> </span>00</span></span> 00 00 00 f5 86 17 71 6d 90 3f b8 42 b5 60 6a</span> \|</span>.......qm.?.B.</span></span>`</span>j</span></span>\|</span> </span></span></span>33</span></span> 5f f1 40 6b 9a 21 d3 00 0a 61 6d 6f 67 75 73</span> \|</span>3_.@k.!...amogus</span></span>\|</span> </span></span></span>2e</span></span> 74 78 74 00 00 00 00 00 00 00 00 00 00 00 00</span> \|</span>.txt............</span></span>\|</span> </span></span></span>00</span></span> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00</span> \|</span>................</span></span>\|</span> </span></span></span>00</span></span> 00 00 00 00 00 81 a4 00 00 00 00 00 00 00 00</span> \|</span>................</span></span>\|</span> </span></span></span>00</span></span> 00 00 00 66 5e 95 f1 67 4e 94 66 cb 42 9b df</span> \|</span>....f^..gN.f.B..</span></span>\|</span> </span></span></span>eb</span></span> af 1b 87 92 ef 0e ec 00 09 74 72 75 74 68 2e</span> \|</span>..........truth.</span></span>\|</span> </span></span></span>74</span></span> 78 74 00 54 52 45 45 00 00 00 06 00 2d 31 20</span> \|</span>txt.TREE.....-1</span></span> \|</span> </span></span></span>30</span></span> 0a 94 c6 2b 91 4b 84 9d 9a 2e 3c 20 e4 3e 93</span> \|</span>0...+.K....</span></span><</span> .></span>.</span>\|</span> </span></span></span>1b</span></span> 69 3f 19 3d bd</span> \|</span>.i?.</span>=</span>.</span></span>\|</span> </span></span></span></code></pre> Nothing too spectacular here, we notice that we see both amogus.txt and truth.txt. We also see TREE at the end of the file, that must be our tree object.</p> However, what happens when we run write-tree</code>?</p> $</span></span> git write-tree</span> </span>aee76412ed220742aeaf02ca1c50519bcea013e1</span></span> </span> </span></code></pre> Let's dump the index again.</p> $</span></span> hexdump -</span>C</span> .git/index</span> </span>44</span></span> 49 52 43 00 00 00 02 00 00 00 02 00 00 00 00</span> \|</span>DIRC............</span></span>\|</span> </span>00</span></span> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00</span> \|</span>................</span></span>\|</span> </span>00</span></span> 00 00 00 00 00 81 a4 00 00 00 00 00 00 00 00</span> \|</span>................</span></span>\|</span> </span>00</span></span> 00 00 00 f5 86 17 71 6d 90 3f b8 42 b5 60 6a</span> \|</span>.......qm.?.B.</span></span>`</span>j</span></span>\|</span> </span></span></span>33</span></span> 5f f1 40 6b 9a 21 d3 00 0a 61 6d 6f 67 75 73</span> \|</span>3_.@k.!...amogus</span></span>\|</span> </span></span></span>2e</span></span> 74 78 74 00 00 00 00 00 00 00 00 00 00 00 00</span> \|</span>.txt............</span></span>\|</span> </span></span></span>00</span></span> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00</span> \|</span>................</span></span>\|</span> </span></span></span>00</span></span> 00 00 00 00 00 81 a4 00 00 00 00 00 00 00 00</span> \|</span>................</span></span>\|</span> </span></span></span>00</span></span> 00 00 00 66 5e 95 f1 67 4e 94 66 cb 42 9b df</span> \|</span>....f^..gN.f.B..</span></span>\|</span> </span></span></span>eb</span></span> af 1b 87 92 ef 0e ec 00 09 74 72 75 74 68 2e</span> \|</span>..........truth.</span></span>\|</span> </span></span></span>74</span></span> 78 74 00 54 52 45 45 00 00 00 19 00 32 20 30</span> \|</span>txt.TREE.....2</span></span> 0</span>\|</span> </span></span></span>0a</span></span> ae e7 64 12 ed 22 07 42 ae af 02 ca 1c 50 51</span> \|</span>...d.."</span>.B.....PQ\| </span></span></span></span></span></span>9b ce a0 13 e1 07 2b 99 c5 b6 0e 3d 53 33 c9 21 \|......+....=S3.!\| </span></span></span></span></span></span>dd b1 75 41 41 84 b1 d8 ec \|..uAA....\| </span></span></span></span></span></span></code></pre> The data following the TREE</code> identifier has increased in size. We will now inspect the contents of this newly written tree.</p> $</span></span> git cat-file -</span>p</span> aee76412ed220742aeaf02ca1c50519bcea013e1</span> </span>100644</span></span> blob f58617716d903fb842b5606a335ff1406b9a21d3 amogus.txt</span> </span>100644</span></span> blob 665e95f1674e9466cb429bdfebaf1b8792ef0eec truth.txt</span> </span></code></pre> Now we seem to have two files inside of the tree.</p> Time to Commit</h2> So now, it's time to create a commit. Above, when we did write tree, we created the tree</code> object aee76412ed220742aeaf02ca1c50519bcea013e1</code>. This includes both of the blobs we made.</p> So how do we commit that? It's actually really simple, we just use git commit-tree</code>.</p> git</span></span> commit-tree aee76412ed220742aeaf02ca1c50519bcea013e1 -</span>m</span> "</span>initial commit"</span></span></span> </span>87a1aa833dccca5ea503e9a7ff81c51fe82c85c6</span></span> </span></code></pre> If we now look at the .git</code> repository, we will see a new object, 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6</code> created. We can verify the type of this new object.</p> git</span></span> cat-file -</span>t</span> 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6</span> </span>commit</span></span> </span></code></pre> Unsurprisingly, it's a commit object. If we look what is inside, we see something that may look familiar to people that regularly use git.</p> git</span></span> cat-file -</span>p</span> 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6</span> </span>tree</span></span> aee76412ed220742aeaf02ca1c50519bcea013e1</span> </span>author</span></span> Christina Sørensen <</span>christina@cafkafk.com></span> 1715792150</span> +0200</span> </span>committer</span></span> Christina Sørensen <</span>christina@cafkafk.com></span> 1715792150</span> +0200</span> </span> </span>initial</span></span> commit</span> </span></code></pre> A fun thing to do now is to try and run git log</code>.</p> $</span></span> git log</span> </span>fatal:</span></span> your current branch appears to be broken</span> </span></code></pre> Seems like our branch is broken hu? Let's fix that. Let's quickly remind ourself of the contents of .git/HEAD</code>.</p> $</span></span> cat .git/HEAD</span> </span>ref:</span></span> refs/</span> </span></code></pre> This doesn't refer to anything. We can easily fix this, let's make a new branch. But what git subcommand will we use this time? None actually, as it turns out, branches are just files in .git/refs/heads/</code> that contain the sha-1 hash of some commit. The name of the file becomes the name of the branch.</p> $</span></span> echo 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6 ></span> .git/refs/heads/main</span> </span></code></pre> Now we just need to switch to that branch. Instead of doing git switch main</code> we can just change the reference in the repository directory .git</code>.</p> $</span></span> echo "</span>ref: refs/heads/main"</span></span> ></span> .git/HEAD</span> </span></code></pre> If you're following along, and you have some PS1 git branch feature, you may have noticed something incredible just after running that command.</p> First, if we run git status</code> now, we see that there no longer are any changes to be committed.</p> $</span></span> git status</span> </span>On</span></span> branch main</span> </span>Changes</span></span> not staged for commit:</span> </span> (</span>use</span></span> "</span>git add/rm <file>..."</span></span> to update what will be committed</span>)</span> </span> (</span>use</span></span> "</span>git restore <file>..."</span></span> to discard changes in working directory</span>)</span> </span> deleted:</span></span> amogus.txt</span> </span> deleted:</span></span> truth.txt</span> </span> </span>no</span></span> changes added to commit (use "</span>git add"</span></span> and/or "</span>git commit -a"</span></span></span></span>) </span></code></pre> Also, we can now run git log</code>!</p> git</span></span> log</span> </span>commit</span></span> 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6 (HEAD -></span> main</span></span>) </span>Author:</span></span> Christina Sørensen <</span>christina@cafkafk.com></span> </span></span>Date:</span></span> Wed May 15 18:55:50 2024 +0200</span> </span> </span> initial</span></span> commit</span> </span></code></pre> Something else that's cool is we can run git log --format=raw</code>, and see output similar to what we got from git cat-file -p</code> on the commit objects sha-1.</p> $</span></span> git log --</span>format</span>=</span>raw</span> </span>commit</span></span> 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6</span> </span>tree</span></span> aee76412ed220742aeaf02ca1c50519bcea013e1</span> </span>author</span></span> Christina Sørensen <</span>christina@cafkafk.com></span> 1715792150</span> +0200</span> </span>committer</span></span> Christina Sørensen <</span>christina@cafkafk.com></span> 1715792150</span> +0200</span> </span> </span> initial</span></span> commit</span> </span> </span>$</span></span> git cat-file -</span>p</span> 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6</span> </span>tree</span></span> aee76412ed220742aeaf02ca1c50519bcea013e1</span> </span>author</span></span> Christina Sørensen <</span>christina@cafkafk.com></span> 1715792150</span> +0200</span> </span>committer</span></span> Christina Sørensen <</span>christina@cafkafk.com></span> 1715792150</span> +0200</span> </span> </span>initial</span></span> commit</span> </span></code></pre> But what about those files that are deleted? We can solve that with git checkout</code> like this.</p> $</span></span> git checkout HEAD --</span></span> amogus.txt truth.txt</span> </span></code></pre> Now if we run git status</code> we get.</p> $</span></span> git status</span> </span>On</span></span> branch main</span> </span>nothing</span></span> to commit, working tree clean</span> </span></code></pre> With this, we have successfully created a commit from scratch.</p> Conclusions</h1> Starting from an empty directory, we have successfully initialized a repository, staged files, created a branch, and committed our work — all without using the standard git commands.</p> Is this efficient? Certainly not.</p> However, by bypassing the porcelain commands, we have exposed the elegant simplicity of git’s plumbing. Understanding how objects, trees, and references interact transforms the system from a black box into a logical machine. This deeper understanding will help you internalize the tool, making complex operations intuitive rather than terrifying.</p> There are still open questions: how does a pull</code> work? How do we manually rebase</code>? That is a topic for another day — or perhaps a therapy session. For now, we will leave them as an exercise for the curious reader.</p> Exercises</h1> What does the hexdump -C</code> look like of the .git/index</code> if we add another blob with hash-object</code> and update-index</code>?</li> Can you discern any patterns from this. What more can you learn about the index file format.</li> What does the zlib compressed data look like inside of the tree object after if we add another blob?</li> What does the .git/index</code> look like after running update-index</code> with a tree blob in the object database?</li> What does the .git/index</code> look like after running update-index</code> and write-tree</code> with a tree blob in the object database?</li> Can you figure out what the myserious f^gNfB</code> means?</li> What does the hexdump -C</code> look like of the .git/index</code> inside a larger project.</li> Can you discern any patterns from this. What more can you learn about the index file format.</li> Does changing the size of the blob from 78 to some other number inside of the zlib compressed data influence git cat-file -s</code>? You will need to decompress and recompress the data from and into the proper zlib format^{[8]</a></sup>.</li> </ul> Footnotes</h1> We don't have to put our git repo in .git</code>. We could use the GIT_DIR</code> environment variable, or the --git-dir=<path></code> flag. ↩</a></p> </li> the zlib wrapper (RFC 1950) — unlike gzip wrapper (RFC 1952) — doesn't store file name and other file system information, which is fine, considering how git manages this elsewhere. ↩</a></p> </li> From this unix.stachexchange answer</a>. Here, we concatenate the gzip magic number and compression method, and concatenate (the actual reason for cat existing) this with the file. We then pipe it into gzip, who can now understand and decompress it. Still, we didn't finish the file with the 8 byte footer, so gzip gets confused, but that doesn't matter, we get to see the data regardless. ↩</a></p> </li> https://git-scm.com/book/en/v2/Git-Internals-Git-Objects</a> ↩</a></p> </li> If you're interested in finding out how the hash is generated, start here</a>. ↩</a></p> </li> Read more herehttps://git-scm.com/book/sv/v2/Git-Internals-Git-Objects ↩</a></p> </li> The mortal enemy of C has many names: empty bytes, null bytes, nop bytes. ↩</a></p> </li> One approach could be to use the hack, with the additional padding at the end, to extract the file. Then, after changing the number, compressing it again, and removing the prepend and appended gzip magic numbers. ↩</a></p> </li> </ol>} Institutional Math Christina Sørensen — Fri, 10 Nov 2023 14:03:00 +0000 Foreword from the future</h3> As 2025 is coming to an end… I’ve been reading over some of my old posts. The tone here is obviously on the more polemic side, but it would be a disservice to my former self to erase the record of my views on academia when I was on the inside.</p> I think it’s worthwhile to conserve not just the more reasoned and polished version of “why did you leave academia” — that I now often find myself regurgitating — but also conserving the “emotion” behind that decision.</p> Because any after the fact rationalization aside, the fundamental decision wasn’t rational, it was emotional. Experienced. Not up for argument. It was an incommensurable experience of a normal science that wasn’t suited to my taste. It was the observation of the fundamental sin in the heart of academia as an institution.</p> If nothing else, then as a warning to my future self to never make the fault of Epimetheus again. Before Newtons birthday seems an apt time to spit on the grave of the enlightenment. So /spit</code>, cheers, and onward!</p> I’ve been wanting to write something terse, dense, with academic heft on this subject a long time. Sadly, I fear I’ll never write such a thing. But I need to speak my heart. I left the mathematical asylum, but I need some closure, and eating Curry isn’t cutting it anymore.</p> Grigori Perelman rejected the fields medal, and the Clay Institute awarded monetary prize for solving one of the “Millenium Problems”. Evariste Galois laid dormant for 10 years after his death in the drawers of Liouville. Gauss and Newton let their personal quibbles turn into abuse of power. Grothendieck left the whole thing behind, burned all his papers. Kaczynski became a domestic terrorist. Lovely field indeed!</p> I’m not interested in money or fame; I don’t want to be on display like an animal in a zoo.</p> – Grigori Perelman</cite></p> </blockquote> The QED project is but ruins — visions of better times. If the proof isn’t surveyable, then it isn’t solvable! Mathematics isn’t just syntactics, no, they’re new age incarnations on the platonic plane of ideas</em>, they’re universal truths</em> of the universe!</em>… Nonsense.</p> Mathematics is circlejerking on tautological systems</strong>.</p> Lockhart can lament all he wants, at least in primary school the problems are trivial, and if you’re done fast you can do something actually useful with the research stipend called public school. At the prestigious academy, the queen of sciences is growing ever more complicated year by year… but ironically, the material taught is the perfect problems for proof systems, we’re teaching students only to replicate. Since the primary concern of these problems is less work for the validator, the true test of your “learning outcome” is your ability to solve solved problems systematically. We must preserve the human validator, by turning said human into a mistaken mockery of the computer.</p> Science, knowledge, the university itself. Open is its doors to any and all. Come research for Boeing and Lockheed, at the faculty of partitioning of 7 hole toruses by means of ballistics, or if it feels too theoretical to indirectly maim and kill people in foreign countries, why not study applied mathematics, and develop social models for companies like Palantir? Profiling, what’s that? Do you not believe in the infallible probabilities?</p> “We need a new Bourbaki” we young people may say to each other, but we all know well that we’d rather get ours and leave this corpse rotting and dying. If anything, what’s needed isn’t more fuel on the fire. Institutional science is systematically pissing in the super soaker wielded by various tyrants, profiteers and oligarchies around the planar graph we all live on. If we are to change mathematics, we should get to the roots of the problem, like we have always done. The foundation on which we built has a crisis, and that foundation isn’t mathematical, it’s social.</p> And for myself, for my mathematical circlejerking?</p> I keep it between friends, private, like good sex. The smut we produce we encrypt and keep safe so as to not show something profane to the plutocrats of the academy. Encrypting — hoping to be kept in those drawers until saner heads emerge to open it.</p> RIME Christina Sørensen — Thu, 09 Nov 2023 16:02:00 +0000 This isn’t a tutorial, it’s more a “here’s what that thing I’ve been working on is about.”[1]</a></sup></p> </blockquote> I like Nix a lot. It’s a so called “obvious good idea”. Even with much of its baggage, some technical, some cultural, it’s probably one of the most interesting things currently happening in software, and you’re making a mistake if you’re not paying attention to it.</p> My desktop, laptop, build server, and even my IRC-bounce server in the “cloud” runs Nix. This makes it easy for me to provision my systems from a single source of version controlled truth. What has made that even better is that all the software projects I maintain include a so called Nix flake, and I make sure to upstream the projects I could imagine people would find some value in to nixpkgs every week on release.</p> One particular project I maintain is called eza, it’s that fork of exa, that you may or may not even be aware that you’re running if you’re a former exa user[2]</a></sup>.</p> Eza relies heavily on Nix. The first way Nix showed up was to help us remove vagrant, a product from HashiCorp, the company famous for making terraform. This was partially because I was uncomfortable with the vagrant business license, but further because Nix could replace it while providing unique benefits. But we do so much more with Nix.</p> The Nix flake sets up the development environment for contributors, in a way that matches the same environment as I myself develop on effortlessly for devs. It helps ensure that commits follow conventional commits, and checks if the code conforms to a battery of linters. It further makes it easy to format all source, and helps build eza in a declarative way.</p> In general, I like to think that we at eza run a tight ship. This is necessary, because we’re working in a surprisingly hard domain, where there is a lot of side-effects — what else would you expect from something so closely tied to the file system.</p> This means that we enforce semantic versioning as well, and generate changelogs that make breaking changes clear.</p> Now with all that in mind, let me quickly explain the lifecycle of a eza version.</p> We start on Thursday, after a release was just made. For most of the week, we slowly and steadily merge pull requests. Then, when Thursday comes, based on whether or not a breaking change was introduced (as is easily parsed from our commit summaries thanks to conventional commits), we determine the next version of eza automatically, and justfile recipes (think make files but less pain) cross compiles binaries for the platforms we support, and cuts and tags the new release, as well as making a “proper” github release.</p> Then, after all that is done, there’s only two things left. First, I run cargo publish</code>, to send the newest eza version to crates.io, from which it is immediately available for download via cargo install eza</code>[3]</a></sup>. Secondly, I update the nixpkgs package (that I, reasonably enough, am the maintainer of), and await someone with commit bit set to merge it.</p> Now, it is often merged quite quickly these days, I think that there is a small group of commiters who’s gotten used to looking for my batch of software updates and merge them without much fuss. But even with a same day merge, due to the nature of nixpkgs, this newest version will actually first be available in some days, usually four to five.</p> This is because the nixpkgs unstable branch lags behind the main branch[4]</a></sup> by a bit. The reasons for why this is are overly technical, but sufficient to say that even while nixpkgs-unstable is the most up to date package distribution in aggregate, compared to arch-extra, which usually has the newest version of eza within a day, for packages like eza, it lags behind.</p> There of course is a potential solution to this, one that I did use for bit. See, since eza has a flake.nix</code> file, one can just refer to that directly in their NixOS config, by making it a flake input (if one does use flakes that is).</p> This is great, that way, you don’t have to wait for nixpkgs-unstable to catch up with our latest release, but there is a catch.</p> Flake inputs have no notion of versioning.</p> Now, if you’ve been keeping your ear to the ground, there has, withing the last few month been two major improvements on this front. Firstly came flakehub, a proprietary platform by many NixOS veterans. Later came the, seemingly spiteful flakestry.dev, by other NixOS veteran. Between the lines, it’s farily clear that this is a direct reaction to flakehub being closed source.</p> The selling point of both of these platforms is the following: you add a github action, and whenever you make a proper github release, that action will push your flake to the respective platform.</p> The problem with both however, at least to me, as someone that distributes software to a fair few amount of users, but even further, just as someone that likes to tinker with my system and loves to make it all neat is the following:</p> People need to add a github action to their repos.</li> Only repos that have done this, and has released after, can actually be used.</li> This only works for github.</li> If this brittle setup breaks in the future, of if releases stop coming, you’ll have to move to a different source.</li> </ul> And I tried to see if these limitations weren’t too annoying for me to use for some months. But aside from the technical limitations, the closed nature of flakehub got to me (at the time, flakestry.dev didn’t exist).</p> Having had this in the back of my head lead to many shower thoughts, which lead to additional shower thoughts, which lead to generally very long showers. Some weeks ago, those shower thoughts collided into a sudden, immediate hacking run, and rime was born.</p> Now, three weeks later, we’re sorta at a place where all the essential features we want are in place.</p> Those things include supporting not just github, but gitlab, codeberg, forgejo, gitea, sourcehut, and even flakehub[5]</a></sup>, supporting semver requirements, whether that be getting any version that is a version $\le 0.3.48$, or just only getting minor and patch updates, but not major version bumps without manual intervention. And like, a lot more features. But really, what makes it great is that I can just specify:</p> {</span> </span> description</span> =</span> "</span>My Configuration"</span></span>;</span> </span> </span> inputs</span> =</span> {</span> </span> eza</span>.url</span> =</span> "</span>http://rime.cx/v1/github/cafkafk/eza.tar.gz"</span></span>;</span> </span> </span> ... your</span> other</span> inputs</span> </span> }</span> </span> </span> outputs</span> =</span> {</span> </span> ... a</span>l</span>l</span> y</span>o</span>u</span>r</span> c</span>o</span>o</span>l</span> o</span>u</span>t</span>p</span>u</span>t</span>s</span> </span></span> </span>}</span> </span>}</span> </span></code></pre> And like that, the eza flake input will always be the latest released (excluding pre-releases) version.</p> This really puts nixpkgs into perspective for me. For projects that already have their own flakes, the value of nixpkgs mainly becomes it being a hand crafted manually updated package whenever there is a new version.</p> And while I’m determined to make eza available to as many users as possible, and having it in nixpkgs will obviously help that… for my own use at least, it’s just so much more neat to have it as a flake input, but without having to include unreleased, potentially unstable changes.</p> It clearly to me also opens some routes to a more decentralized NixOS landscape, since it adds a sort of layer between the latest raw flake and the users computer, reminding me of the layer of snow that build up when you place something sturdy in a hostile, cold climate with a lot of snow(flakes). A layer called rime.</p> Although much of the work has been done by Gergely Nagy</a>, who I’ve been able to successfully nerdsnipe. ↩</a></p> </li> Since distributions shim in eza in the place of exa nowadays. ↩</a></p> </li> Although, sadly, it’s impossible to include man pages and completions via crates.io. ↩</a></p> </li> Technically, the “master” branch, but let’s not indulge that anachronism. ↩</a></p> </li> And it’s written to be easily extendible, so adding new forges is trivial. ↩</a></p> </li> </ol> Adding dollar sign (apply) to nix Christina Sørensen — Thu, 03 Aug 2023 14:34:33 +0000 This post is very low quality for comedic effect.</p> </blockquote> A detailed list of instructions</h2> Take one nix</a>, apply this patch</a>[1]</a></sup>:</p> What does this do? It adds a new token $</code> to the nix languages parser and exposes it as __dollar</code>, essentially by copying the way this works for __lessThan</code>.</p> Hack on your nix:</p> nix develop </span>./bootstrap.sh </span>./configure $configureFlags --prefix=$(pwd)/outputs/out </span>make -j $NIX_BUILD_CORES </span>outputs/out/bin/nix repl </span></code></pre> Now in the nix repl</code> that just opened:</p> nix-repl> __dollar = x: y: builtins.foldl' (a: b: a b) x y </span></code></pre> This makes $</code> an infix operator that’s essentially just a generic apply.</p> Now just create a toy example to demonstrate:</p> nix-repl> concat = x: y: z: (x + y + z) </span>nix-repl> concat $ ["literally" " " "dollar sign"] </span></code></pre> And your repl will say:</p> "literally dollar sign" </span></code></pre> Tadaa! That’s all it takes to add the $</code> operator to nix, just like Haskell.</p> The end $\text{\(^ヮ^)/}$</p> diff --git a/src/libexpr/parser.y b/src/libexpr/parser.y </span>index 201370b90..4b26af623 100644 </span>---</span> a/src/libexpr/parser.y </span></span></span>+++</span> b/src/libexpr/parser.y </span></span></span>@@</span> -353,6 +353,7</span> @@</span> void yyerror(YYLTYPE * loc, yyscan_t scanner, ParseData * data, const char * err</span> </span></span></span> %left AND </span> %nonassoc EQ NEQ </span> %nonassoc '<' '>' LEQ GEQ </span>+</span>%nonassoc '$' DOL </span></span> %right UPDATE </span> %left NOT </span> %left '+' '-' </span>@@</span> -411,6 +412,8</span> @@</span> expr_op</span> </span></span></span> \| expr_op LEQ expr_op { $$ = new ExprOpNot(new ExprCall(makeCurPos(@2, data), new ExprVar(data->symbols.create("__lessThan")), {$3, $1})); } </span> \| expr_op '>' expr_op { $$ = new ExprCall(makeCurPos(@2, data), new ExprVar(data->symbols.create("__lessThan")), {$3, $1}); } </span> \| expr_op GEQ expr_op { $$ = new ExprOpNot(new ExprCall(makeCurPos(@2, data), new ExprVar(data->symbols.create("__lessThan")), {$1, $3})); } </span>+</span> \| expr_op '$' expr_op { $$ = new ExprCall(makeCurPos(@2, data), new ExprVar(data->symbols.create("__dollar")), {$1, $3}); } </span></span>+</span> \| expr_op DOL expr_op { $$ = new ExprOpNot(new ExprCall(makeCurPos(@2, data), new ExprVar(data->symbols.create("__dollar")), {$3, $1})); } </span></span> \| expr_op AND expr_op { $$ = new ExprOpAnd(makeCurPos(@2, data), $1, $3); } </span> \| expr_op OR expr_op { $$ = new ExprOpOr(makeCurPos(@2, data), $1, $3); } </span> \| expr_op IMPL expr_op { $$ = new ExprOpImpl(makeCurPos(@2, data), $1, $3); } </span>diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc </span>index ddf529b9e..00622a142 100644 </span>---</span> a/src/libexpr/primops.cc </span></span></span>+++</span> b/src/libexpr/primops.cc </span></span></span>@@</span> -3583,6 +3583,24</span> @@</span> static RegisterPrimOp primop_lessThan({</span> </span></span></span> .fun = prim_lessThan, </span> }); </span> </span>+</span>static void prim_dollar(EvalState & state, const PosIdx pos, Value * * args, Value & v) </span></span>+</span>{ </span></span>+</span> state.forceValue(args[0], pos); </span></span>+</span> state.forceValue(args[1], pos); </span></span>+</span> // pos is exact here, no need for a message. </span></span>+</span> CompareValues comp(state, noPos, ""); </span></span>+</span> v.mkBool(comp(args[0], args[1])); </span></span>+</span>} </span></span>+</span> </span></span>+</span>static RegisterPrimOp primop_dollar({ </span></span>+</span> .name = "__dollar", </span></span>+</span> .args = {"e1", "e2"}, </span></span>+</span> .doc = R"( </span></span>+</span> Just like haskell :) </span></span>+</span> )", </span></span>+</span> .fun = prim_dollar, </span></span>+</span>}); </span></span>+</span> </span></span> </span> /************************************************************* </span> * String manipulation </span></code></pre> ↩</a></li> </ol> The Pull-Request Heist Christina Sørensen — Thu, 03 Aug 2023 14:34:33 +0000 Your effort to remain what you are is what limits you.</p> – The Puppet Master</cite></p> </blockquote> The Beginning</h2> May 3, 2014. Benjamin “Ogham” Sago has just made his first commit to Exa</a>. This will go on to be one of the most used ls alternatives, with ~22k stars, 655 forks, and 151 contributors.</p> For almost a decade, he maintains exa. Yet, around 2021, activity is drastically slowing. Eventually, the project falls dead.</p> With 63 open pull requests, 276 issues, and some major bugs, the project is on its deathbed. The users are begging</a> for activity.</p> At this point, other projects are being advertised in the issues for various features. Features that — mind you — are already finished in the pull-requests, but without anyone to merge. The situation is dire, are we all doomed to use LSDeluxe</a> and endure its awful color scheme?[1]</a></sup></p> That’s where I somehow came in.</p> The rabbit hole yak-shave</h2> The terminal emulator kitty is really neat. So is exa, and I’ve been using exa for a fairly long time. Never have I given it much more thought than installing it from nixpkgs, I’m for the most part blisfully unaware of the projects state (which speaks to it’s quality no doubt).</p> Kitty has a feature. You can type ctrl+shift+e</code>, and it highlights links with letters, a bit like vim easy-motions, and allows you to effortlessly open them in your browser.</p> As a keyboard only extremist, this pleases me.</p> Of course, the thing is, when I exa some directory, I can’t “easy motion” for files. And I want that. So I look into it. Turns out, kitty has that feature! And so does ls!</p> ls --hyperlink=auto </span></code></pre> Neat!</p> Then, tragedy… exa doesn’t. What! A bit of searching reaveals to me a PR on exa. It’s been there since April. It’s awsome!</p> But it isn’t merged. Why? I orient myself in the issues. I try to figure out who to complain to. But there is none, the project looks dead.</p> I dip my toes in some forks, see if I can get this done. Seems trivial. Then, my partners has to help sister move. I have the weekend free. The nerd snipe tinglies are coming on. A hacking run</a> is imminent — whether I like it or not.</p> Onto the programming</h2> Fork other forks, we’ll write our own! Call it eza, old habits die hard, and the x is next to the z on my keyboard. This will suffice. In one bold move we have already solved half of the hard problems of computer science[2]</a></sup>, I’d say that’s a good start.</p> Now we just have to get those pull request over in our repo. One way to do this is the following:</p> git fetch \ </span> git@github.com:ogham/exa.git/pull/<pr-id>/head:<target-branch> </span></code></pre> This way, we can take a pull request branch from exa, and copy it into a branch in our repo. The problem is, there’s 63 pull-requests!?! We’d have to do this for each of them, push the branch to github, then slowly click through the webui and make that pull request, one at a time.</p> As a keyboard only extremist, this displeases me.</p> There is no way we’re doing this. If only we had a list of all those repo-ids, we’d be able to at least automate the fetching.</p> Then, lightbulb. Github has a REST-API</a> ripe for our queries[3]</a></sup>. A bearer token is created. An absolute unit of a curl request is crafted. The world pauses for a moment as we lay the fate of the project into the wire.</p> curl -L \ </span> -H "Accept: application/vnd.github+json" \ </span> -H "Authorization: Bearer <the-bearer-token>" \ </span> -H "X-GitHub-Api-Version: 2022-11-28" \ </span> 'https://api.github.com/repos/ogham/exa/pulls?per_page=100' > open-prs-full.json </span></code></pre> Then we wait for the response.</p> After the Suspense</h2> Well. It worked.</p> cat open-prs-full.json \| head -n 10 </span></code></pre> [</span> </span></span> {</span> </span></span></span> </span>"</span>url"</span></span></span>:</span> </span>"</span>https://api.github.com/repos/ogham/exa/pulls/1221"</span></span>,</span> </span></span></span> </span>"</span>id"</span></span></span>:</span> </span>1446083658</span>,</span> </span></span></span> </span>"</span>node_id"</span></span></span>:</span> </span>"</span>PR_kwDOATIZtM5WMXxK"</span></span>,</span> </span></span></span> </span>"</span>html_url"</span></span></span>:</span> </span>"</span>https://github.com/ogham/exa/pull/1221"</span></span>,</span> </span></span></span> </span>"</span>diff_url"</span></span></span>:</span> </span>"</span>https://github.com/ogham/exa/pull/1221.diff"</span></span>,</span> </span></span></span> </span>"</span>patch_url"</span></span></span>:</span> </span>"</span>https://github.com/ogham/exa/pull/1221.patch"</span></span>,</span> </span></span></span> </span>"</span>issue_url"</span></span></span>:</span> </span>"</span>https://api.github.com/repos/ogham/exa/issues/1221"</span></span>,</span> </span></span></span> </span>"</span>number"</span></span></span>:</span> </span>1221</span>,</span> </span></span></span> .</span>.</span>.</span> </span></span></span></code></pre> Was this what we needed? We do see a number</code> property there, and if we look in the html_url</code> property, we see that it matches the expected url for that pull request. Now here is the part where I’d shown off my jq prowess, but I have no such thing. So instead, I make do with some regex.</p> cat open-prs-full.json \ </span> \| rg -i 'pulls/\d+"' -o \ </span> \| tr -d "pulls/" \ </span> \| uniq \ </span> \| tee pr-ids.txt \ </span> \| wc -l </span></code></pre> 63 </span></code></pre> The number of lines is right, but is the output?</p> cat pr-ids \| head -n 5 </span></code></pre> 1221" </span>1219" </span>1210" </span>1204" </span>1202" </span></code></pre> Ahh, close enough, a little adjustment and we are there:</p> cat open-prs-full.json \ </span> \| rg -i 'pulls/\d+"' -o \ </span> \| tr -d "pulls/" \ </span> \| tr -d '"' \ </span> > pr-ids.txt </span></code></pre> Now did we get it?</p> cat pr-ids.txt \| head -n 5 </span></code></pre> 1221 </span>1221 </span>1219 </span>1219 </span>1210 </span></code></pre> Seems we did!</p> The Part Where We Jam Those Pull-Requests into GitHub</h2> At this point, the estrofem tablet under my tongue had dissolved, and I’d woken from my groggy morning state. A purely performative crack of the knuckles were in order, as I leaned over my desk like a hawk that’s caught sight of a mouse.</p> If you’re a keyboard-only extremist like me, web-ui is a nuisance to be solved. GitHub has gh</code>, or the GitHub CLI. It is extremely useful to sync your nixpkgs fork, and send your package definition pull requests. But today, old reliable was gonna learn some new tricks.</p> Then I paused. And stared.</p> In my head flashed visions of old Bell Labs informational videos</a> on the Unix Operating System. The voice of Brain W. Kerninghan echoed in my head — “I think the notion of pipelining is the fundamental contribution of the system…”. The hacking cgi was front and center as I said out loud “It’s a UNIX system, I know this!”. Temporarily absolved of my mortal coil, I resided in waves of abstraction and shell, time and language lost their meaning to the beauty of the baud. It was as if every bit moved in concert with my thoughts, as if the collective will of every electron in my sillicon rock filled out the sails of my spaceship gently surfing on the stream of raw information entering and exiting me. As I slowly raised my cup of tea to my mouth, not for one moment breaking this intimate emergence and embodiment of me in my code, in my will, in my craft, before me it appeared, at the end of time and space itself. The final destination, the one truth of programming, the zen was ever closer. Then the tea — hotter than a NVidia GPU running KDE — removed my taste buds faster than a Arch maintainer removes system critical boot code. What had been my ticket to enlightenment turned into a leaky abstraction all over my desk.</p> However, I had tried so hard and gone so far, there was no way that in the end, it wouldn’t really matter.</p> Now — armed with a more modest disposition, xargs, and the GitHub CLI — I went to work. Soon, emerged the command that would seal my fate:</p> cat</span></span> pr-ids.txt \ </span></span></span></span> \|</span> xargs</span></span> -</span>I</span>%</span></span> -</span>n1</span> sh -</span>c</span> \ </span></span></span> '</span>git fetch git@github.com:ogham/exa.git pull/%/head:pr-%; git checkout pr-%; git push; gh pr create --title \'</span></span>(exa PR</span></span>) %</span></span>\'</span></span></span> --</span>body</span> \'</span>github.com/ogham/exa/pulls/%</span></span>\'</span>'</span> </span></span></span></code></pre> As a smut novel once said[4]</a></sup>: There comes a time in every woman’s life where she reaches a crossroads. A chance to choose between two paths. One is unknown, filled with potential and struggles that demand growth and change, but could lead to new beginnings. The other is familiar, grooved with well-worn ruts and established twists, muddy holes and isolation. The path that, while lonely, fits like a glove.</p> When — if — I hit enter, I’d be down the road of committing myself to maintaining the damn thing. Were I not to, I’d never downlive the internal shame. Were I to do it, I’d have allocated quite a substantial amount of time in my life to pesky users, stubborn contributors, and the chore and dull drag of changelogs, version bumps, and releases.</p> </p> As I hit enter, and as the pr tab on the repo slowly grew in the web interface, I hoped this wouldn’t be a mistake.</p> The Aftermath</h2> Over the next two days, me and sbatial</a> would go onto hack on eza. When monday came, 63 had been reduced to 18.</p> Hyperlinks, SE-linux contexts, git repo statuses, human readable relative dates, more security fixes than I have fingers to count them on. And the darn grid bug, the pesky bugger had been swatted and was now but a stain on the git history.</p> With flakes, CI, and sheer will, we had overcome. Trying out eza was as imple as running:</p> cargo install eza </span></code></pre> Or if one was so inclined:</p> nix run github:cafkafk/eza </span></code></pre> But had we won?</p> The battle, no doubt, but not the war.</p> Reflections on DevOps and Infra</h2> In the excellent talk at Chaos Communication Conference 36: From Managerial Feudalism to the Revolt of the Caring Classes</a>, the late David Graeber summarizes a core problem in modernity as eloquently as only he could:</p> [Whenever] I say, talk to a Marxist theorist, wherever, they try to explain value, which is what they always like to do, they always take the example of a [bottle].</p> […] Well, look at this bottle, you know. [It] takes a certain amount of socially necessary labor time to produce this, say it takes, you know, this much time, this much resources. It’s always about production of stuff.</p> But a teacup or a bottle. […] You produce a cup once, you wash it like 10000 times. Most work isn’t actually about producing new things, it’s about maintaining things. […]</p> So I think that what we need to do is we need to start over. We need to realign. First of all, think about [work], not as producers, but as carers.</p> </blockquote> </p> I spend four years of my life studying “the queen of the sciences”, mathematics herself. One view that I particularly enjoyed were that of the toolmaker. Sure, we were studying abstract structures, but that’s being a student. A mathematician was a tool maker.</p> That view might have a stink of applied mathmatics, but I think it’s true even for those who’d call themselves pure[5]</a></sup>.</p> This too is true in DevOps, in systems programming, in infrastructure. At the end of the day, we’re toolmakers.</p> In my mispend youth, in the grips of startup culture, the idea of the elusive 10x developer, while just a mirage was still very alluring. Regardless of the obviously wrong and problematic discurse related to it, there is no doubt that some people move mountains, and others don’t. I wanted to move mountains.</p> What I’ve later came to learn is that’s just an illusion. The idea of the lone genious, burning the midnight oil is just a myth. John Carmack, Linus Torvalds, George Hotz, they’re just a personality cult.</p> And this isn’t just some clever insight I’ve come to, it’s stolen</a>. But I like my iteration on it.</p> A certain Archmage of Infrastructure has the right idea about being a “force multiplier”. What really matters to move these mountains is to enable those that want to move them. The most dedicated cyclist in the world will cycle less than an average bikesmith will enable others to do.</p> In this creating of tools, and in this care work of maintaining them, we are force multipliers, and in our multiplication of force, the impossible becomes possible.</p> This is the point of DevOps. This is the point of infra. This is the next level of abstractions hackers should strive for if they wanna boil the ocean.</p> Post-Credit Scene</h2> From 5:00-19:00, for two days, I had hacked. 3 meals a day, hydrating, and a walk in nature had supported this endeavor[6]</a></sup>.</p> At last, a knock on the door, my partner had come home. After a hug and some pleasantrise, came the question:</p> So, what did you do with your weekend?</p> </blockquote> I eagerly typed eza</code> in my terminal emulator, and showed how there were now letters next to the entries, and clicking them copied the text.</p> What should have been pure awe was a look of slight concern, followed by a shoulder shrug and a kiss on the forehead.</p> That’s all you did while I was gone?</p> </blockquote> It’s all love, I bet it’s just a bad screenshot on their git repo <3 ↩</a></p> </li> See the first quote here</a>. ↩</a></p> </li> And we’d never scrape a website, that’s rude :^) ↩</a></p> </li> “Arabella Stone, Nashville’s darling, is eager to shuck her prim-and-proper rep, and a few wild months spent checking items off her ’ summer bucket list’ is the way to do it. First up: kiss the man she’s crushed on since she wore a training bra, the bad boy of country himself, Blue bassist Charlie Tucker. For Charlie, a beautiful woman flirting with him isn’t out of the norm—but a beautiful woman bolting after the hottest kiss of his life sure is. And when he finds out his kiss-and-run Cinderella is none other than Arabella Stone, daughter of his label’s CEO, he knows he’s in trouble. Because not only is she a Stone, she’s also his employee for the next few months at the recording studio he just bought. Over the course of one thrilling summer, Arabella and Charlie chip away at her bucket list and fight the simmering attraction between them…knowing that once it’s all over, so is their time together.”</p> Steamy. ↩</a></p> </li> Which really should be called theoretical. ↩</a></p> </li> All this rise and grind never sleep never eat never enjoy your life stuff is stupid. If you wanna be even like mildly consistent, you gotta get this right. Dumb stuff like hacking is like the last thing you get to in mazlows hierachy[7]</a></sup> for a reason. ↩</a></p> </li> Yes, I know it’s “debunked”, it’s a working model geez louise. ↩</a></p> </li> </ol> Warts and all Christina Sørensen — Mon, 24 Jul 2023 16:10:10 +0000 Learning</h2> The better you get the worse you get. The more I learn the more I learn that I don’t know much.</p> This can be overwhelming at times. I think it’s pretty normal to feel this way. But I also see it can easily become a limiting type of negative feedback.</p> Looking back at old projects, old essays, old anything, I see flaws. I was so naive. Simultaneously, I feel like I see how my output has gotten better.</p> But realizing this negative feedback loop — as in keeping with the theme — just makes you even more insecure! Because, if this model is true (and it generally seems to be), then this post I’m writing right now will seem so bad to me in a year… not to imagine 10 years!</p> Knowing this, it’s easy to fall into perfectionism. There’s always something to improve after all. But this is dangerous. Because, while it might be the case that in 10 years, I can easily see all the flaws, right now I can’t. And it seems to be my experience that no matter how much I correct, no matter how much time I spend trying to make something perfect, I only realize what was actually missing much later.</p> Worst, things don’t even make it out the door. Perfectionism can kill projects. When that happens, I don’t realize the flaws. The projects I never finished because I didn’t know how to explain it the right</em> way or didn’t know just how to say what I wanted to say perfectly, those I still don’t know how to improve.</p> Granted, they might just still be out of my reach, and thus simply never get finished because of this, but I feel everything I do is out of my reach, even when doing it. And then it’s done, and I hate it a little. And then a year goes by, and I hate it a lot.</p> I have to finish things to improve. Incomplete things are in a constant state of improvement. They’re too maleable to be at fault[1]</a></sup>. They’re beyond critique because they’re meerly in progress, they’re not solid enough to grasp and criticise.</p> Ideally, I shouldn’t get discouraged by feeling stupid. In fact, I’m starting to feel stupid, and I’m kinda glad about that, because I know how much I had to learn to be this dumb. But I dislike everyone’s complex with wanting to be the wisest of all the G(r)eeks by pretending that they don’t know anything.</p> Knowing that you know nothing is a paradox, and a platitude.</p> He who knows, does not speak. He who speaks, does not know.</p> – Lao Tsu</cite></p> </blockquote> This is a dumb quote. It should really be:</p> Those who speaks, does not know. Those who knows, does not speak, and plateaus.</p> </blockquote> You can’t know things, you can only know them more. So knowing, isn’t a destination, but a process. But it’s a painful one.</p> Self Redaction</h2> I feel like contemporary culture expects a self redaction from the individual of their history. You’re not supposed to fail in public. In a spectacle, there are just dunkers and those that are dunked on. Being wrong is death of your socially distributed ego.</p> What this false internet reality leaves out is the numerous failures that lie at the end of everything even remotely sort of cool. Without this ever in the public eye, the culture perpetuates and intensifies perfectionism in us all.</p> I think that to some extend, this isn’t just a moral issue, but also an individual one. The idea of being unable to fail in public means you can’t grow in public either.</p> This I find limiting.</p> Further, for those around us, the network effects of being imperfect seem to be worthwhile. In fact, I’d argue that in a time of flawless, but false public personas, being not just a finished product but a evolving history is a much more potent and intersting identity to embody. And it’s one that anyone with any sense should find preferable to the superficially flawles.</p> I feel like I’ve already made numerous public mistakes, and I don’t see that slowing down, and maybe I shouldn’t want that, each one is a sign of me going in the right direction.</p> I don’t know how to end this.</p> Reference implementations and their consequences have been a disaster for the human race. ↩</a></p> </li> </ol>

Quantizing Gemma 4's MTP drafters

Christina Sørensen — Fri, 29 May 2026 00:00:00 +0000

A while after Gemma 4 came out, Google released a set of Multi-Token-Prediction (MTP) drafters for it. Pairing a large model (the verifier) with a tiny MTP drafter (the assistant) enables speculative decoding: the small model guesses the next few tokens, and the big model verifies them in one parallel pass. When it works, the throughput gain is most of the way to free.

To put it shortly: MTPs are one of the rare free lunches we get with LLMs. Or well, on paper, like with free lunches, that’s the assumption.

These are notes on getting that working locally. The previous post</a> was about why this should have been easier than it was.

If you try to convert the Gemma 4 MTP drafter weights using mainline ggml-org/llama.cpp</code>, the converter rejects them:
Model Gemma4AssistantForCausalLM is not supported </code></pre>Mainline supports the base Gemma 4 models, but it does not know what a gemma4_assistant</code> is. The converter lacks the class, and the runtime lacks the MTP API. The runtime PR #18886</a> has been in draft for months. A companion converter PR #22738</a> was opened and closed unmerged with one of four tasks done. The working path is a fork: ikawrakow/ik_llama.cpp</code></a>, specifically the feat/gemma-4-mtp</code> branch from PR #1744</a>. The resulting GGUFs only load in that fork. Downstream wrappers — Ollama, LM Studio, jan.ai, llama-cpp-python</code> — will all reject them until they backport the changes. This is the cost of running new architectures the week they ship: you swap inference backends to match the model, not the other way around. Drafters are tiny. The E2B drafter’s MTP head is 78M parameters; the 31B’s drafter is roughly 470M; the 26B-A4B’s is about 408M. Their job is to predict tokens accurately enough that the verifier accepts them, so acceptance rate dominates everything else. Aggressive quantization causes the verifier to reject more drafts, and the pipeline becomes slower than running the verifier alone. The quantization policy for drafters is the inverse of what you’d do for a large base model: Do use: bf16</code>, f16</code>, Q8_0</code>, Q6_K</code>, Q5_K_M</code>, Q4_K_M</code>.</li> Do not use: IQ1_*</code>, IQ2_*</code>, Q2_K</code>, Q3_*</code>. These cost more rejected drafts than they save in load time.</li> </ul> imatrix</code> calibration is not available for these drafters at the moment. PR #1744 builds the gemma4_mtp</code> drafter graph with a hardcoded GGML_ASSERT(has_target_ctx)</code>, so standalone llama-imatrix</code> runs abort in llama_decode</code> before producing anything. The IK fork’s signature quants (IQ4_KS</code>, IQ4_KSS</code>, IQ5_KS</code>) derive their precision-per-bit advantage from imatrix-guided scale selection, so without an imatrix they collapse to roughly K-quant quality at the same bit budget. Shipping them under those labels would be misleading. Quantizing directly from bf16</code>/f16</code> to Q4_K_M</code>/Q8_0</code> is the realistic option until the tooling catches up. At ≥4 bits the precision gap from missing the imatrix is small — single-digit percent on most benchmarks. At Q3 and below it would matter, which is the other reason Q3 isn’t a sensible target here. One empirical surprise: on a 4060 Laptop GPU, pairing unsloth’s Gemma-4-E4B-it-Q4_K_M.gguf</code> verifier against both a Q8_0</code> and a Q4_K_M</code> drafter at --draft-max 3</code>, the Q4_K_M</code> drafter beat the Q8_0</code> by ~13% in total throughput. The smaller drafter’s faster draft step outweighed the acceptance-rate cost. The “always pick the highest-bit drafter” rule assumes you are memory-bandwidth-rich and compute-poor; on a laptop GPU you are neither. Benchmark your own silicon. llama-quantize</code> silently embeds the absolute on-disk paths of your imatrix file and calibration corpus into the GGUF metadata. There is no flag to disable this at quantize time. The *-f16.gguf</code> intermediate is clean; only quantized files have the leak. Inspect any quantized GGUF and the two offending keys are sitting there: nix-shell -p python3Packages.gguf --run \ 'gguf-dump --no-tensors model.gguf' </code></pre> Key</th> What it leaks</th></tr></thead> quantize.imatrix.file</code></td> Absolute path to the imatrix file on the build host</td></tr> quantize.imatrix.dataset</code></td> Absolute path (and filename) of the calibration corpus</td></tr> </tbody></table> Uploading the raw output of a quantization pipeline publishes your infrastructure layout — usernames, ZFS pool names, directory structure — alongside the weights. The fix is gguf_new_metadata</code> from the Python gguf</code> package, which rewrites the file with selected keys removed without touching the tensors: nix-shell -p python3Packages.gguf --run ' for f in /srv/models/<MODEL>-GGUF/*.gguf; do [ "$f" = "*-f16.gguf" ] && continue python3 -m gguf.scripts.gguf_new_metadata \ --remove-metadata quantize.imatrix.file \ --remove-metadata quantize.imatrix.dataset \ --force \ "$f" "$f.clean" done ' </code></pre> Each rewrite runs at ~5–15s per GiB and shrinks the file by exactly the size of the two strings removed (≈190 bytes). Verify before uploading: gguf-dump --no-tensors model-clean.gguf | grep -v INFO </code></pre> kv_count</code> should be exactly 2 less than the original. The imatrix.*_count</code> integer fields stay (those carry no path information). A smoke test confirms the rewritten file still loads: llama-completion -m model-clean.gguf --jinja \ -p 'The capital of France is' -n 20 --temp 0 </code></pre> Coherent output means the rewrite didn’t corrupt anything. Sanitized quants in hand, the next step is pairing. A drafter is not a standalone language model: it has to be paired with a verifier that shares its exact vocabulary. For Gemma 4 that vocabulary is 262,144 tokens. The size of that vocabulary is also the reason the MTP speedup applies to structured output. The drafter sees the same <|tool_call|></code>, <|tool_response|></code>, and <|channel|></code> tokens as the verifier, so tool-calling agents and structured-generation workflows benefit on the full response, not just the natural-language parts. The vocabulary itself isn’t quantized — it lives in a separate GGUF section. A working ik_llama.cpp</code> invocation: llama-server \ --model gemma-4-31B-it-Q8_0.gguf \ --model-draft gemma-4-31B-it-assistant-Q8_0.gguf \ --spec-type mtp \ --draft-max 3 \ --draft-p-min 0.0 \ -ngld 99 \ --n-gpu-layers 99 \ --ctx-size 32768 \ -ctk q8_0 -ctv q8_0 \ -b 1024 -ub 1024 \ --jinja </code></pre> The relevant levers: --spec-type mtp</code> enables the speculative-decoding path from PR #1744.</li> --model-draft</code> is the drafter GGUF; its vocabulary must match the verifier.</li> --draft-max 3</code> sets the speculative chain length. Values from 1 to 4 are all reasonable; --spec-autotune</code> will probe and pick one for your workload.</li> --draft-p-min 0.0</code> accepts all drafts; raising it shortens chains on difficult prompts.</li> --jinja</code> applies the model’s chat template, including Gemma 4’s tool-call format.</li> </ul> PR #1744’s reference numbers, 31B verifier + 31B drafter at Q8_0/Q8_0 on a data-center GPU: Run</th> Throughput</th> Acceptance</th></tr></thead> Baseline (no MTP)</td> ~21 t/s</td> —</td></tr> MTP --draft-max 1</code></td> ~35 t/s</td> ~89%</td></tr> MTP --draft-max 2</code></td> ~44 t/s</td> ~83%</td></tr> MTP --draft-max 3</code></td> ~49 t/s</td> ~74%</td></tr> MTP --draft-max 4</code></td> ~49 t/s</td> ~64%</td></tr> </tbody></table> A bit more than 2× throughput, after all of the above. The throughput gains are real, and on the right hardware they are substantial. The work it takes to surface them is also real: undocumented CLI divergences, an obscure fork, manual metadata sanitization, quantization rules that invert the usual advice. None of this is in the model card or the upstream README. The path to a working setup is reconstructed from PR threads and stack traces. The next post</a> in this series takes the result and runs it on a 2016 Xeon with no GPU, which is where the command line stops being a paragraph. Open weights are not open source Christina Sørensen — Tue, 26 May 2026 00:00:00 +0000 I’ve been quantizing Gemma 4 and it’s MTP drafters recently. The next two posts are about how that went. This one is about why it was so much work. When a lab releases a model under an open license, what they almost always release is the weights. Sometimes a tokenizer. Occasionally a short technical report. Rarely the training data, data-processing code, training code, evaluation harness, or the recipes used to produce the published quantizations. People still call this open source, and I think that’s misleading. If you can’t rebuild it, you can’t fork it. If you can’t fork it, the word doesn’t mean what it used to. Calling it open is like calling an executable binary open because you can execute it. Unless you consider assembly code open source, I think it’s really obvious why this misses the mark, and is exellent marketing spin. The people releasing models don’t seem to feel responsible for whether anyone can run them, or have good results. Its more like box ticking. Here it is? There is no shared vocabulary for how a model should be packaged, or which inference engine it expects, or what quantization recipe was used to make the file on whatever hub they pick. The settings that actually decide whether a model is usable on a given machine — mlock</code>, KV-cache dtype, flash attention, MoE routing, draft model pairing — live in PR descriptions and Discord screenshots. The model card lists the benchmark scores and links a HuggingFace space. So you download the weights, get a result that’s slow or incoherent or both, and you don’t know whether the model is bad, your quantization is bad, your runtime is bad, or you’ve just configured something wrong. Most people, reasonably, conclude it’s just their hardware, or that open models don’t actually work for normal users. The tools that have grown up to paper over this make the problem worse, not better. Ollama is the obvious example. It picks a quantization, runtime, context and batch sizes for you, and doesn’t tell you what it picked or why. When the output is bad, there’s no thread to pull. You can’t tell if you’re holding it wrong because the thing was designed so you can’t hold it at all. It’s just a blackbox! This wouldn’t matter much if the defaults were good. They aren’t. A default Q4_0 of a small model with no imatrix calibration and a mismatched chat template will be much worse than the same model run carefully, and the user has no way of knowing the difference exists. AND NO ONE REALLY TALKS ABOUT HOW MUCH THIS MATTERS! Concretely: a while after Gemma 4 came out, Google released a set of Multi-Token-Prediction drafters for it, designed to make the base models faster via speculative decoding. The obvious thing to do is download the GGUF and run it under llama.cpp</code>. That doesn’t work, and the reasons it doesn’t work are a fair sample of the genre: Mainline llama.cpp</code> doesn’t know what a Gemma4AssistantForCausalLM</code> is. The converter rejects the drafter outright. To convert it, you have to find ikawrakow/ik_llama.cpp</code>, on a feature branch, from a PR that hasn’t merged.</li> llama-imatrix</code> aborts on that branch’s MTP graph because the build hardcodes an assertion that a target context is bound. So you skip imatrix calibration, and your Q4 quants are noticeably worse than the ones every other model on the hub ships with.</li> llama-quantize</code> writes the absolute on-disk paths of your imatrix file and calibration corpus into the GGUF metadata. There’s no flag to turn this off. If you upload without scrubbing, you publish your home directory layout next to the weights.</li> </ul> None of this is in the model card. None of it is in the upstream README. You find it by trying things and reading stack traces. No one does this. Seriously. I mean yes. Part of it is that they just use what the people that did bother to do it made, fair. But there’s like 4 quants for this on huggingface at the time of writing, one is mine. FOUR! For the state of the art MTP/MOE models! The hardware to run a 30B-class MoE at reading speed isn’t exotic. A five-year-old workstation with enough RAM will do it. The bottleneck isn’t the silicon, it’s everything between the weights file and a working configuration: which fork, which quant, which flags, which template, which version of which template. Ohh, and getting it all running, good luck unless you’re… literally an infrastructure engineer? Most people give up somewhere in the middle and go back to the API, and I don’t blame them. I do think the labs releasing these models could make this much less bad than they have. Publishing the imatrix corpus would help. Publishing the quantization recipe would help. Picking one reference runtime and keeping the model working on it would help. Treating the deployment story as part of the release, rather than something the community is expected to reconstruct, would help. Until that changes, “open source” for language models mostly means “someone else has to figure out how to run it, and none of the freedoms that used to mean open source are actually present”, and the people best positioned to figure it out are charging by the token. I’ll write up the actual quantization steps — which flags, which traps, what the metadata leak looks like, how to scrub it — and what it took to run the result on a 2016 Xeon with no GPU, in two follow-up posts this week. Thanks for reading :) Another NixOS Election Cycle Christina Sørensen — Thu, 02 Oct 2025 00:00:00 +0000 And here I thought we’d have a boring election cycle</h2> If you’ve not been following NixOS governance, recently the moderation team decided to all collectively resign. I’ve been informed that I’ve been one of the people that certain steering committee members have wanted to moderate harder, specifically because I’ve been calling out their undisclosed conflicts of interest, and providing actual level headed, direct criticisms that they’ve yet managed to answer. This is despite their alleged pledge to respond to community concerns. One of the people I’ve been led to believe specifically has been pushing for aggressively moderating me has yet to respond to any messages I’ve sent them in private, which to me is even more odd, they’re not rude, and could likely have cleared up any legitimate moderation issues. I think that, looking back at the previous steering committee, and even watching their talks at NixCon, it was quite clear that most of them were not really interested in governance. Paraphrasing, most of them said a version of “I’m not the most competent person to be here”, and those were the ones that at least have the insight to make that realization. I think that many were there simply to add a new line to their resume. What does it mean for me running?</h2> Nothing. I’ll run. If enough people vote for me, I’ll do what I set out to do in my candidate statement. If I’m not elected, I’ll put my focus elsewhere, and make myself available to any elected Steering Committee members on policy and governance questions, just like the last cycle. I don’t think that the Steering Committee is the only (or even the most effective) way for me to improve the project and ecosystem. But it is an option I’d be foolish to throw away. And I have much I want to do. Here is the start. Process</h4> I’ve read the Steering Committee retrospective from Gabriella</a>. To me, the challenges raised here are of primary concern. Currently, the steering committee isn’t leading the community. And if the final instance required for decisive action in the project is indescisive, then what’s the point? So that’s my primary concern. The steering committee should actually get things done. I think we can solve this. Transparency</h4> Meeting notes and employers of board and steering committee memeber should be made public - to the extend possible. There should be public logs of both. Everything should default to being documented in public, including decisions and how members vote. If the steering committee finds certain subjects too sensitive, they may be redacted, but even then, the fact that a vote or discussion took place should be documented. Who voted for and against - even if we can’t say for what - should be written down for everyone to see. The steering committee must become auditable and accountable to everyone in the project. Unblocking Adoption</h4> Flakes aren’t unstable. That’s mainly a naming convention at this point. It is a major disadvantage for the project, and the official installer adds friction. Flakes should be installed by default. Defaults should be sane. Many decisions in the project aren’t made based on what’s best for the project, but where measures to alleviate immediate harm. Now those same measures have themselves become harmful. By engaging with the community, and listening for places where their colleagues are blocked or choosing alternatives to official tools, I’ll correct any blockers with the steering committee and community, and ensure we become the best choice. I think Nix is a paradigm shift in how we do computers, and I want to convince the rest of the world of that. I want to work with Nix for the rest of my career, and for that to become a reality, I’m running for the steering committee again. A final question: Will you go the distance</h2> I know a lot of people are scared that they’ll end up voting for someone that will turn passive and be checked out. I am scared of voting for such people. It was a major pain in the last election, that even if I was aligned with a candidate and they got elected, they just didn’t actually push for anything, and seemed to essentially give up. If that’s something you’re scared will happen, I think specially if you’re a long time community member, the best proof I can give that it wont happen if you vote for me is that… I’m still here. I’d not want to waste your vote. But I’d also like to temper your expectations. I can only do so much. If you vote for a bunch of people that are not gonna go the distance, then my primary role becomes keeping you updated on why the process remains broken. I can be a voice for accountability, but I cannot be the entire committee. Tracking Sattelites with Consumer Components Christina Sørensen — Sun, 05 Jan 2025 06:00:00 +0000 A Global Network of Ground Stations</h2> At 38c3</a> I attended a workshop on building a sattelite ground stations for communicating with LoRa based sattelites, typically used for research or weather probes. Generally speaking, these are so called “cubesats”</a>, which are small, relatively inexpensive sattelites usually built up of off the shelf components in a modular fashion. In fact, it turns out that with one of these inexpensive sattelites, a small launch would actually only cost in the range of $50,000-$100,000, not a small amount, but still, a lot more accessible than what you would initially imagine. Combine this with LoRa (literally Long Range) — a proprietary radio communication technique — and you have the components for a globe spanning network of sattelites and amatur ground stations. This has lead to a suprising amount of these sattelites currently in low-earth orbit around the globe</a>. Typically these aren’t geostationary (meaning located over the same point on the planet at all times), and so, it is useful for e.g. a smaller university to have access to a globe spanning network of sattelite stations. TinyGS</h2> Enter TinyGS, tiny ground stations</a>. TinyGS is an open network of Ground stations distributed across the globe, run by amateurs. It allows operating LoRa based weather probes, sattelites, and other “flying objects”, using consumer electronics. The basic idea is that you get an ESP32 as well as a some sort of LoRa module, an antenna, and flash the tinygs firmware onto it. Then you just connect it to the tinygs servers using MQTT, and you will be able to opearte your board on the tinygs website, as well as being able to contribute to the global sattelite tracking effort. Build your own Ground Station</h2> Hardware</h3> If you wanna build your own, the easiest possible route is just to acquire a LoRa32 V2.1_1.6 - 433MHZ CH9102F</code> board. Usually you can find these online, and make sure that you get one with a small antenna. You’ll need to flash the firmware, the easiest way is to use the TinyGS web flashing tool</a>, and sorry, but you’ll need to use a chrome based browser for this. Setup</h3> What’s worse, you’ll also need to install telegram, and create an account, as the TinyGS projects communications mainly happen there, and that means the registration bot for your new board also lives on telegram. Join the TinyGS telegram group here</a>, and then start a conversation with the tinygs personal bot here</a>. Also, make sure you join the TinyGS community chat in the group first, as the bot will require you to be a member before signing you up. When you are in that chat, send the /start</code> command to the bot for some info about it’s usage. You’re mainly just interested in the /mqtt</code> command, which gives you your credentials for the mqtt server. Now with your credentials, and your participation in the TinyGS community (hi!), you’ll want to plug in your flashed board. After a few moments, it should start it’s own WiFi AP, that you can connect to. On the screen of your board, it should show the IP address that you’ll then want to connect to on the boards WiFi. This will give you a configuration inteface. Here, you enter the configuration, and then configure a ground station name, create an admin password (to be used with the admin</code> account), enter the WiFI SSID and Password that you want the board to use for internet connectivity, your lattitude and longitude (this doesn’t have to be exact, but should be vaguely correct withing 25km~ because it is used to determine what sattelites to try and connect to), and a timezone. Further, you’ll want to add your MQTT credentials from the telegram bot. If you’ve done everything correctly, you’ll receive a message on telegram from the bot, like I did in the above screenshot. You’re now on the network. Start operating your sattelite</h3> After you’ve setup your station, you’ll want to /login</code> with the bot to be able to operate it fully. This will give you access to even more information about your device, and the ability to edit it’s description and image, and to control how it is tracking sattelites, or setup manual tracking if you wanna just track something specific, or you don’t like the automatic selection. Below that, you’ll find a log of the packets you have recieved so far. Notice here that the red circle indicated a CRC checksum error, meaning you recieved a faulty package. Also notice that there are two packets with a CRC error that were caught by just one station (meaning just your station). In this case, if you click on the packet, you’ll get the text that: Unrecognized packet. This packet does not match the known structure of this satellite. It might be an unknown packet or just terrestrial noise. </blockquote> However, for those that don’t have a CRC error, that’s a real sattelite packet you’ve caught. Well done!. If you click on it you can even read the details about what you caught. For example, the HOD-HOD-1A</code> capture from above provides the following information when you click on it. As you can see, it shows what the content of the packet was, both as text, hexadecimal, and parsed json that you can expand and browse through. Below that, you’ll see a list of other stations that caught the packet. The view to the side shows the geographical location of the stations that caught the sattelite communication. I wanna capture more packets!</h3> As you will likely realize after running your station for a few days, you’re probably not capturing a lot of packets. This is because the default antenna doesn’t really perform well, and likely, you’ll also ideally want to make sure that your station is outside, and isn’t obstructed, and that weather conditions are favorable (anecdotally, I also tend to capture much more packets at night). I’ll not elaborate more on this in this post, as it is something I’m working on myself, but I may follow up later with a separate post on what it takes to increase the amount of packets you capture. In short, you’ll want a 1/4-wave ground plane antenna. Obligatory Conclusion</h2> With inexpensive consumer hardware, it is possible to build your own sattelite ground station, and join a globe spanning network of space enthusiasts capturing sattelite data. That is pretty cool :3 Conventional Commits Change my Brain Christina Sørensen — Wed, 24 Jul 2024 00:00:00 +0000 What is a conventional commit?</h2> In case you’re not aware, conventional commits</a> is a standard format for writing commit summaries. It looks like this. build(deps): bump git2 from 0.18.3 to 0.19.0 Bumps [git2](https://github.com/rust-lang/git2-rs) from 0.18.3 to 0.19.0. - [Changelog](https://github.com/rust-lang/git2-rs/blob/master/CHANGELOG.md) - [Commits](https://github.com/rust-lang/git2-rs/compare/git2-0.18.3...git2-0.19.0) --- updated-dependencies: - dependency-name: git2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> </code></pre> This example is from dependabot</a>, which does conform to conventional commits. Notice that only the first line, the commit summary, is part of the conventional commits spec here. The last line is a “footer”[1]</a>, or technically, a trailer</a> added by the git commit -s</code> flag, -s</code> short for --signoff</code>. The exact spec looks like this: <type>[optional scope]: <description> [optional body] [optional footer(s)] </code></pre> Unlike many project specific standards, such as that of angular</a> or nixpkgs</a>, what sets conventional commits apart is that it aims to be a universal commit standard, a bit like how semantic versioning</a> seeks to be a universal software versioning standard. In the same sense as semantic versioning provides meaning through version numbers, such as a major</code> number increasing meaning a breaking change, conventional commits also seeks to encode information about the state of a project in the commit summaries, and provides enough information to reasonably tell when it’s time to do a patch, minor, or major version bump. It further also provides ways to specify if a commit contains a breaking change, via a git footer. It may be easiest to explain this by just showing you the spec. The commit contains the following structural elements, to communicate intent to the consumers of your library: fix: a commit of the type fix</code> patches a bug in your codebase (this correlates with PATCH</code></a> in Semantic Versioning).</li> feat: a commit of the type feat</code> introduces a new feature to the codebase (this correlates with MINOR</code></a> in Semantic Versioning).</li> BREAKING CHANGE: a commit that has a footer BREAKING CHANGE:</code>, or appends a !</code> after the type/scope, introduces a breaking API change (correlating with MAJOR</code></a> in Semantic Versioning). A BREAKING CHANGE can be part of commits of any type.</li> types other than fix:</code> and feat:</code> are allowed, for example @commitlint/config-conventional</a> (based on the Angular convention</a>) recommends build:</code>, chore:</code>, ci:</code>, docs:</code>, style:</code>, refactor:</code>, perf:</code>, test:</code>, and others.</li> footers other than BREAKING CHANGE: <description></code> may be provided and follow a convention similar to git trailer format</a>.</li> </ol> Additional types are not mandated by the Conventional Commits specification, and have no implicit effect in Semantic Versioning (unless they include a BREAKING CHANGE). A scope may be provided to a commit’s type, to provide additional contextual information and is contained within parenthesis, e.g., feat(parser): add ability to parse arrays</code>. </blockquote> So basically, if there is a type fix</code>, that’s a PATCH</code> bump to your semver, a feat</code> is a MINOR</code> bump, and a BREAKING CHANGE:</code> footer or a !</code> after the type/scope is a MAJOR</code> bump. If we for instance have commit with the summary fix: change windows-only imports to be windows-only</code>, that’s a PATCH</code> change, and if we are on version 3.2.48</code>, then next version will have to be at least 3.2.49</code>. If we instead had a summary feat: add --no-|show-symlinks flags for filtering output</code>, then that would a MINOR</code> change, and again, if we were on 3.2.48</code>, next release would be 3.3.0</code>. Lastly, if we had a summary feat(flags)!: add --classify=always,auto,never</code>, even thou it has a feat</code> type, the !</code> indicates this is a breaking change, and so we must cut a new MAJOR</code> version on next release if we include this commit, meaning that if we were on 3.2.48</code>, we’d go to 4.0.0</code>. Yes. Breaking changes means a new major version. Yes, that might seem sever to you if you’re not used to being at the helm of something being used by a lot of people, but that’s how we make these sausages (at least if we’re actually trying to do it correctly). The guy who authored the semver spec, Tom Preston-Werner</a> (also known for co-founding GitHub</a> and creating gravatar</a>) has a blog post about this called Major Version Numbers are Not Sacred</a>. What people end up doing in practice is building up a lot of breaking changes and then releasing them all at once, to not constantly destroy the API/ABI and UX of their software. How this ties in with a more continuous development is a very interesting question, e.g. how does eza</a> manage to release a new version every week, yet also work on breaking changes like config files or moving to a new command line parser (specifically clap</a>). I think such a discussion is far out of scope for this post, but I may write about it in the future. Conventional Commits and Linguistic Relativity</h2> If you were to ask wikipedia what “Behavioral Design” means, it would reply: Behavioural design is a sub-category of design, which is concerned with how design can shape, or be used to influence human behaviour. </blockquote> Now, how is that relevant to us as developers, maintainers, or just hobbyist hackers? Well you see, the design of conventional commits, whether intentional or not, has a lot of decisions that end up shaping how contributors interact with your project. When I initially forked eza</code>, I mostly moved us to conventional commits fairly early because I was concerned with a maintenance burden. I knew that exa</code> had been a project of maintainer burnout, and what I was trying to do was create longevity for the project. That meant that if there was something I could offload to someone or something else, I would, including changelogs. And yes, conventional commits means I don’t have to write changelogs, I can automate it, and it means I don’t have to think too hard about what version is next, it’s automatically generated. In fact, I can release eza every week because a release is just running two scripts, doing a cargo publish</code>, and a nix-update eza</code> in the nixpkgs</a> repo and opening a PR. While that release velocity is great, I found another, much more interesting perk of using conventional commits. Namely that conventional commits lead to better commits, better commit summaries, better PRs, and better structured commits. For instance, if a project just allows a commit summary to be an arbitrary string, what you’ll often end up with is either something like made extensions work</code>, a single commit that encompasses vast refactorings of the codebase, introduces a feature, fixes 3 different bugs, and is one big monolithic and unrevievable mess. On the other hand, it also saves you from the series of wip</code>, wip 2</code>, formatted code and added quantum cryptography</code>, made compile</code>. These are equally unreviewable, and completely destroy your git log. Now, you may have the discipline of projects like nixpkgs</a> of asking contributors to rebase their commits, so that they each represent unique atomic changes. But even then, what conventional commits gives you is a language to express what these changes actually are. Before I had conventional commits as something I could just write without looking at the spec, I would genuinly struggle putting words to what my changes did. It made it hard to create good PRs to projects. Now, it’s much easier. I just go: refactor(parser): used enums instead of bools in match logic</code></li> fix(parser): capital letters not differentiated</code></li> fix(parser): long lines ignored</code></li> feat(parser): add long flags</code></li> docs(parser): document long flag usage</code></li> style: change flag rendering</code></li> ci: check parser flags workflow</code></li> </ul> Not only does this make my PR much more structured, but by getting into that habit, I find that my work also becomes much more organized. Instead of just working on some hodgepodge mix of things at the same time and committing either to save my work or to put all of it together in a disorganized sack of a single commit, I find that I am much more able to dissect a problem into discrete solvable steps. I’d even go as far as saying this makes you a better programmer, as breaking down a large problem into small solvable chunks is a great problem solving strategy. What conventional commits does is it gives you this language to describe the steps it takes to solve a problem, and getting familiar with that vocabulary not only makes you a better programmer, but it makes it much easier to work with you, and it heavily reduced the burden on maintainers. Linguistic relativity is the idea that the structure of a language influences the speakers world view and cognition, changing how you interact with your surroundings. Linguistic relativity is a hypothesis, but I find it pretty easy to prove the weak case that if you can’t signify something, you will struggle to operate with it. In other words, conventional commits get you merged faster, and makes your code better. Hopefully you’re sold by this point. There are some tools for dealing with these, such as git-footers</a>. Personally, I don’t really use those. But it’s good to know they exist. ↩</a> </li> </ol> rustls and aws-sdk seems incompatible with FOSS Christina Sørensen — Mon, 17 Jun 2024 14:00:00 +0000 I should preface this with the fact that I am by no means a lawyer or anything like it. Nothing here constitutes a legal argument or should be taken as true. I may be completely wrong or have missed crucial details. As part of a rust project I’ve been working on at dayjob, I’ve had to audit all dependencies for license compatibility with the AGPLv3, our preferred license. Sadly, I encountered what I think is a major ecosystem blocker, and we will likely not be able to license the code under AGPLv3, or any GPL license for that matter, having to go with a more permissive, but compatible license until the problem this post is about is fixed. Which is a shame, and something I wanna bring more attention to. A fairly fundamental crate in rust is ring</a>, providing “safe, fast, small crypto using Rust”. To understand how fundamental it is, consider that other crates such as rustls</a>, the aws-sdk-rust</a> crate, and sct</a> make use of ring. And this is by no means an exhaustive list. The problem with ring, as you may well notice if you dare read its rather convoluted license</a>[1]</a>, is that amongst it’s licenses is the OpenSSL license. The problem with the OpenSSL license is it’s equivalent to a BSD4 clause license. That is, it contains the problematic advertisement clause. * 3. All advertising materials mentioning features or use of this * software must display the following acknowledgment: * "This product includes software developed by the OpenSSL Project * for use in the OpenSSL Toolkit. (http://www.openssl.org/)" </code></pre> This clause is well known</a> for being GPL-incompatible, and indeed, if you look up the OpenSSL license on the gnu.org</a> list of gpl compatible licenses</a>, it is indeed the case that the license is considered GPL-incompatible. But so what? Big deal? Big deal… yes. This means that if you want to interface with s3 storage[2]</a>, or use rustls, you’ll be GPL incompatible! For example, consider the following GNU Affero General Public License Version 3 projects that use crates in the aws-sdk</code> suite, or rustls</code>. zed</a></li> Signal-Calling-Service</a></li> garage s3</a></li> quickwit</a></li> rustdesk</a></li> </ul> And these are just some quick examples I found with a quick github search, there is probably much more software currently out there in the rust ecosystem that’s using a license that its crates are fundamentally incompatible with[3]</a>. While this is a problem the ring</code> crate seems to be aware of</a>, it’s sad to see that there is such a large wound on the FOSS compatibility of the rust crate ecosystem. Not to put any blame anywhere. I am well aware that if this was easy to fix, it wouldn’t be a problem anymore, and anyone wanting to criticize the ring devs should spend the effort helping them fix the situation, rather than complaining, they seem eager to solve this. But it is something I think needs more awareness. Update 2024-06-19: We’ll try to go with EUPL-1.2, as it is compatible with all BSD licenses. Also, it seems to have many other useful properties. This does not make the problem disappear however, but it does create a path forward for a stronger license for rust projects affected that want SaaS loophole protections. This is a problem the ring devs are working on. Let it be a lesson to start your project off using REUSE</a> or paying the price down the line >:3 ↩</a> </li> Yes, there are other crates than the aws-sdk, but when investigating this for the project in question, none seem to be tennable solutions. Vet your dependencies! ↩</a> </li> Yes, there are potential ways to mitigate this, but by a quick skim, I haven’t seen any of those implemented in these projects, and implementing them would likely be both architecturally ugly, and horribly inefficient. For… reasons, I’ll not actually elaborate on how you could potentially circumvent licenses like this, because I’m really not a lawyer. ↩</a> </li> </ol> Merge or Rebase Christina Sørensen — Wed, 29 May 2024 13:40:00 +0000 This isn’t the article that’s gonna explain merge vs rebase for people that haven’t used git before (sorry). Rather, this is gonna be about some more meta things for maintainers, release engineers, and other people who care about development process. Git gives you a lot of options. How you use it isn’t prescribed in advance. There are a lot of ways that people might use git differently, but here, I’m more seeking to investigate some of the ways of using git that are relevant to people in charge of git repositories with a non-trivial amount of contributors, like myself. While I enjoy having these things done differently between projects I’m working in and part of maintaining — as it gives some diversity to the methods I get to use in my work, and let’s me continuously evaluate multiple methods[1]</a> in practice — I do think it makes sense that for a single project, there is consistency in the way it’s developed, so that expectations can be set, and procedures can be learned and become second nature. There are two main questions for a project to decide when it comes to merging via git. How to merge feature branch changes from PRs.</li> How to merge newest changes into outdated branches.</li> </ol> I’ll discuss both, and then summariese my observations. Merging branches</h2> When some developer starts working on a software feature, especially in open source, those changes, typicall, will be made in a separate branch that has branched off from the main branch. This is often called a feature branch. This allows the developer to work on their feature in isolation, and, at some point, put those changes back into the main branch, for all people to use. This is what some may call trunk based development, although the lines get fuzzy about how long a PR should live. That discussion is out of scope for this post, but I’d love to follow up with a post about that at some point. When moving changes from your feature branch into main, you can have potentially three possible options for merging. Fast Forward</li> Rebase and Fast Forward</li> Merge</li> </ul> For fast forward, main may not have changed while you were working, and so you can “fast forward” your changes on top of main, without having to worry about any conflicts. This is, for both developers and maintainers, extremely easy, with some reservations that will be discussed later. Another option is to take the changes from the updated main branch, and git pull --rebase origin main</code> them into your feature branch, placing your changes on top of the latest changes from main. As you may guess, this allows you to git merge --ff-only</code> your changes into the main branch. The final strategy is the classical merge. You may decide to merge your changes, and the changes in main together, using a special commit that represents the operation of merging the two together, a merge commit. Such a commit has special properties, such as having two parents, and being the result of the unification of two separate branches. Of course, if you’re not doing just a simple fast forward, either a rebase-ff or merge may require that you solve potential code conflicts. For instance, someone may have changed a line that you have also changed, leading to the two versions colliding. For the merge, you resolve this by making the merge commit define the merged version, for a rebase, you’ll go through your changes and adjust them so they fit cleanly onto the branch you’re pulling. One important difference to note for us, is between the two, merging into main typically gets done by the maintainer, where as rebasing gets done by the developer working on the feature branch. Rebasing also has some downsides. For example, if the person rebasing doesn’t have the PGP private key of the people who’s commit are included in the rebase, they cannot sign the commits. This means that if Alice made 3 commits and signed them with her PGP signature, Bob can’t later rebase those changes onto some other commit without breaking that signature. This is because, while the code doesn’t change, the commit revision does, and so the signature is wrong. Merging does not have this issue, as a merge is simply the actual merge commit, containing the new state of the branch, that potentially specifies how to rearrange everything so as to not cause any conflicts. Thus it doesn’t suffer from breaking the signatures. Now, if you’re using rebase for merges into main, as we do in eza, you may still want a signature verifying that a trusted person has indeed verified these changes are up to whatever standard they guarantee (eza at best guarantees that it runs on my linux box that you should be able to install it on NixOS[2]</a>). The way we solve this is by tagging all our releases with a signed tag, making sure you know that I’ve personally said that this is the codebase I want you to run for a given release. But even then, it’s coping. It’s not as nice as having a tree full of the signatures of everyone that’s committed, right on their commits. Another point raised against rebases is they rewrite history. This is true, unlike a merge that just merges two branches and keeps both in history, the rebase will make it appear as thou the feature branch was developed on top of the main branch. I don’t personally think this is a huge concern, except the loss of potential pretty git log --graph</code> outputs. One of the annoyances of merges, in comparison, is that they’re very noisy. Consider this output from nixpkgs: git log --oneline bfb7a882678e Merge pull request #314109 from trofi/githooks.tests-fix-eval 47bdef656cc5 Merge pull request #311260 from purepani/update-svelte-language-tools fc165a03b23a Merge pull request #309524 from r-ryantm/auto-update/enet 3de810d52cbe Merge pull request #309724 from r-ryantm/auto-update/secp256k1 1d6a2f5a4d4d Merge pull request #309787 from r-ryantm/auto-update/python311Packages.minio a36fa5451c44 Merge pull request #309426 from Rconybea/add-sphinxcontrib-ditaa 890f8e436a56 Merge pull request #309476 from r-ryantm/auto-update/goperf dfa36c1d67f5 Merge pull request #312557 from r-ryantm/auto-update/intune-portal 7d8ed5ce921d Merge pull request #291853 from greaka/grafana d0a20d7c5955 Merge pull request #314112 from khaneliman/bicep 4d2462511f06 Merge pull request #314099 from mrkline/snapper-and-borgbackup-doc-fix 3e3ac0e7baa8 Merge pull request #305516 from OPNA2608/init/lomiri/ayatana-indicator-display a46ce7c77d9d svelte-language-server: convert to buildNpmPackage 7962cbb2326b Merge pull request #314082 from superherointj/bicep-0.27.1 5771dbfa7d2f bicep: fix updater script 2db4e7d035f8 Merge pull request #313729 from pluiedev/zhf-24.05/gobang 44744fc83f3c githooks.tests: fix eval 0f3add331c6a segger-jlink: 794l -> 796b </code></pre> This… isn’t nice, and makes history way harder to read. Compare to eza: git log --oneline 22dbeec2 (HEAD -> main, origin/main, origin/HEAD) build(deps): bump libc from 0.2.154 to 0.2.155 82e4f0f3 build(deps): bump trycmd from 0.15.1 to 0.15.2 bbede6d4 (tag: v0.18.16) chore: release eza v0.18.16 70891fa1 docs(README): use 3 columns for packaging status badge e8e0b6da fix: change windows-only imports to be windows-only 15528d8b docs(install): fix typo in `INSTALL.md` 26e5943a docs: update INSTALL.md 20241984 build(deps): bump DeterminateSystems/nix-installer-action from 10 to 11 feff970d docs(man): replace decay with color-scale a1c36389 build(deps): bump DeterminateSystems/flake-checker-action from 5 to 7 7437c2a1 (tag: v0.18.15) chore: release eza v0.18.15 f1ef455b feat(devtools): add optional tag argument to deb-package.sh 23502d3c fix(devtools): correct command for latest tag in deb-package.sh 292cf7ec feat(devtools): return to original commit at the end of deb-package.sh 4f542eaf docs(reaedme): add some keywords for benefit of ctrl-f 6d6612d2 docs(readme): move heading out of collapsed section abe9f587 docs(readme): correct heading levels in markdown 01a9c2aa docs(readme): add how to find man pages in terminal and online. Partly fixes #967 f78e4bb6 (tag: v0.18.14) chore: release eza v0.18.14 01919cdd build(deps): bump palette from 0.7.5 to 0.7.6 414f70ba build(deps): bump unicode-width from 0.1.11 to 0.1.12 1554472a build(deps): bump libc from 0.2.153 to 0.2.154 bebd39c0 build(deps): bump uzers from 0.11.3 to 0.12.0 df7d51fc feat: add icon for "cron.minutely" directory 8b7dc5f5 (tag: v0.18.13) chore: release eza v0.18.13 c0df8ecd feat: generate completion/manpage tarballs on release 8afb5cc2 (tag: v0.18.12) fix: checking for deref flag in file_name a4782d1d feat: add scheme filetype and icons 87b36785 fix: allow unused imports for freebsd 99562e3a (tag: v0.18.11) chore: release eza v0.18.11 07f67708 fix: build aarch64, arm without libgit2 17733e9a fix(netbsd): enable the rule only for NetBSD. 7664a1fb ci: bump NetBSD version to 10.0 462fc344 fix(netbsd): fix clippy lints 75f1f8cf (tag: v0.18.10) chore: release eza v0.18.10 </code></pre> Isn’t that a lot nicer? I think so. Of course, what may be lost is that it’s no longer obvious from what PR a change originated. It also isn’t obvious to me when some feature branch was created, how long it was worked on and so on. This isn’t… really that important either, but it’s nice to have in logs. But you can just look at branches if you need to know this for tooling or something. For nixpkgs, you get a to have a visual overview of how branches end up in main. It looks like this: git log --oneline --graph * 4d4571b20a29 (HEAD -> devpi-loadcredential) nixos/devpi-server: fix loading credentials as DynamicUser * bfb7a882678e Merge pull request #314109 from trofi/githooks.tests-fix-eval |\ | * 44744fc83f3c githooks.tests: fix eval * | 47bdef656cc5 Merge pull request #311260 from purepani/update-svelte-language-tools |\ \ | * | a46ce7c77d9d svelte-language-server: convert to buildNpmPackage * | | fc165a03b23a Merge pull request #309524 from r-ryantm/auto-update/enet |\ \ \ | * | | 8e729c70a119 enet: 1.3.17 -> 1.3.18 * | | | 3de810d52cbe Merge pull request #309724 from r-ryantm/auto-update/secp256k1 |\ \ \ \ | * | | | f96a90834c5e secp256k1: 0.4.1 -> 0.5.0 * | | | | 1d6a2f5a4d4d Merge pull request #309787 from r-ryantm/auto-update/python311Packages.minio |\ \ \ \ \ | * | | | | 24d453186792 python311Packages.minio: 7.2.6 -> 7.2.7 * | | | | | a36fa5451c44 Merge pull request #309426 from Rconybea/add-sphinxcontrib-ditaa |\ \ \ \ \ \ | * | | | | | 197d6261ebf8 python311Packages.sphinxcontrib-ditaa: init at 1.0.2 | * | | | | | 51de558f5634 maintainers: add rconybea * | | | | | | 890f8e436a56 Merge pull request #309476 from r-ryantm/auto-update/goperf |\ \ \ \ \ \ \ | * | | | | | | 8eea66b3f69a goperf: 0-unstable-2023-11-08 -> 0-unstable-2024-05-10 * | | | | | | | dfa36c1d67f5 Merge pull request #312557 from r-ryantm/auto-update/intune-portal |\ \ \ \ \ \ \ \ | * | | | | | | | 010c4a334eaf intune-portal: 1.2404.23-jammy -> 1.2404.25-jammy * | | | | | | | | 7d8ed5ce921d Merge pull request #291853 from greaka/grafana |\ \ \ \ \ \ \ \ \ | * | | | | | | | | 254dbdcc6296 grafanaPlugins.grafana-oncall-app: init at 1.5.1 | * | | | | | | | | 0e5f44658ee6 maintainers/team-list: add fslabs | * | | | | | | | | 8d6f8c9ed75a maintainers: add lpostula | * | | | | | | | | 7bda925dacb2 maintainers: add greaka * | | | | | | | | | d0a20d7c5955 Merge pull request #314112 from khaneliman/bicep </code></pre> Another problem worth considering is that a rebase may — as Astrid</a> pointed out — results in main being in a state where a git bisect</code> of a build command may fail, if you haven’t ensured that all commits in the feature branch still worked after being rebased on top of the latest main. Now this is rare, but rare isn’t really a good argument for not caring here, as breaking once would be extremely annoying for people working on a bugfix, leading to false git dissect</code> positives. There are solutions to this. You could (perhaps) have each commit in a branch run against CI. This may be outside of your compute budget thou. Another option is to run your build command, e.g. cargo build</code> on each commit after a rebase, but then, this kind of checking step usually falls on the maintainer, and we, as maintainers, should seek to limit our own burdens as much as possible (to prevent burnout[3]</a>, the killer of most FOSS projects). To eza, this isn’t fatal, but a very interesting point worth keeping in mind. Updating a branch</h2> Of course, since there may be multiple people all working on separate branches, there is another problem. One person might have worked on something, at the same time as you worked on something else, and then gotten their changes merged before you had a chance. But if that happens, your changes might now conflict with what was in the main branch when you started, and worse, their changes may make ruin your work, or even make it unable to compile! The first choice a project should make, is whether or not to allow outdated feature branches to be merged. The problem with tolerating outdated branches is you will not be able to fix any problems related to integrating the code with the main codebase before merging it. This puts the burden of ensuring the code works on the maintainer, rather than the developer, and this is bad! If you do choose to not allow this, another question faces you: how should the main branch be merged into the feature branch when the feature branch is out of date with the main branch. Your options are either to merge the main branch into your feature branch, or to rebase your changes on top of main, making it like if you had started working on the latest changes. To me, it’s a more simple dilemma, merges shouldn’t exist if they don’t meaningfully represent merging two things together. Simply updating your branch is what rebase does best, and having merges both from feature branches into main and from main into your feature branch is absolutely tasteless. This clutters your history for no reason. It also makes it harder for you to fix your commits when the reviewer tells you that you had a faulty commit 3 commits before you merged main into your feature branch and kept working for 4 more commits. In general, rebasing with merge commits is tedious, and we only consider merges viable for the main branch under the premise that we won’t later need to rebase main, because everything that gets into main is in a state of being up to standard. Sadly, code forges lack a feature to only allow updating branches with the projects preferred standard, unlike for merges where rebase, squash, and merge are all options, that all can be disabled by the repositories owner(s). Summary</h2> So what where the tradeoffs? Merging feature branches into main Merges. Preserves history.</li> Gives you git log --graph</code>.</li> Doesn’t break PGP signatures.</li> Creates ugly git log</code>.</li> Gives weak protection that all commits build (if you have CI)</li> Puts burden of verification on maintainer.</li> </ul> </li> Rebase. Rewrites history.</li> No git log --graph</code>.</li> Breaks PGP signatures.</li> Creates useful git log</code> output.</li> Needs more care if you want all commits to be validated.</li> Puts burden of verification on committer (mostly).</li> </ul> </li> Fast Forward. Like a rebase, but with less drawback.</li> </ul> </li> </ul> </li> Updating your feature branch. Merges. merges make git rebase -i</code> difficult, when asked to change your commits.</li> It makes your git log</code> unreadable.</li> Doesn’t provide any benefits.</li> Mostly just a quirk that git even allows this, it doesn’t make sense.</li> </ul> </li> Rebase. The preffered choice.</li> Fits like a glove for updating your feature branch.</li> </ul> </li> </ul> </li> </ul> As a maintainer, remember that what where really trying to do if offload any non-maintainer burdens onto relavant roles, like developers and reviewers. We may wear multiple hats, often we are the final reviewers, but even then, it shouldn’t be our job to ensure a clean merge, we should meerly verify it. The main goal is always: maintainers shouldn’t need to fix the feature branch! This also applies for formatters. I personally mostly use alejandra for nix, but dayjob uses nixfmt (the RFC 166 version). I just hit a format button before commiting anyways, so it’s just cool to see the different options and feel out which ones I truly prefer. ↩</a> </li> Because I don’t have any MacOS, Windows, or BSD boxes to test on, and we don’t have any maintainers that do with the spoons to be involved in giving such guarantees. ↩</a> </li> Consider reading this excellent resource https://opensource.guide/maintaining-balance-for-open-source-maintainers/</a>, and this mozilla worksheet https://docs.google.com/document/d/1esQQBJXQi1x_-1AcRVPiCRAEQYO4Qlvali0ylCvKa_s/edit?pli=1#heading=h.xd8n2v3f4866</a>. ↩</a> </li> </ol> Grok Git Repos Christina Sørensen — Wed, 15 May 2024 00:00:00 +0000 Introduction</h1> Git is often perceived as a necessary evil, a tool to be tolerated rather than mastered. But treating your primary work tool as a mystery is a vulnerability. When the abstraction leaks — and it always does — the developer who understands the internals can surgically repair the damage. In contrast, those who rely on surface-level knowledge often leave behind a sloppy timeline. A history riddled with vague messages and botched merges signals a lack of care that code quality alone cannot hide. Unfortunately, we often learn git through folklore — memorizing obscure commands to save us when things go wrong, without understanding why they work. We rely on luck rather than logic. To escape this ritualistic dependency, we must strip away the interface. To truly grok git, we will ignore the high-level commands and build a repository from the ground up, proving that the magic is actually just simple, elegant engineering. Creating a Git Repository from Scratch</h1> Let's create a folder for us to work in. $ mkdir grok $ cd grok </code></pre> Now, we have a folder. A folder isn't a git repository, as we can verify. $ git status fatal: not a git repository (or any of the parent directories): .git </code></pre> As the status</code> subcommand tells us, neither is the directory we currently created, of any of it parent directories a git repository. If that wasn't the case for you, that would be because you created the folder grok</code> as a subdirectory of a git repository. </blockquote> The error seems to mention a .git</code> folder. Turns out, when speaking of a git repository, we are actually referring to this folder in most cases[1]</a>. So, in theory, it should be simple enough to create a git repository right, just create a .git</code> folder and we are good to go? $ mkdir .git $ git status fatal: not a git repository (or any of the parent directories): .git </code></pre> Ohh, apparently not so. If we were to look up the definition of a repository in the git glossary</a>, we would see the following. A collection of refs together with an object database containing all objects which are reachable from the refs, possibly accompanied by meta data from one or more porcelains. A repository can share an object database with other repositories via alternates mechanism. </blockquote> Ahh, simple. Seems like we need to have some refs and an object database with all the objects refs can reach. Also, there might be meta data from porcelains? Although the definition may seem complex, the underlying concept is straightforward. Rather than summarizing it briefly, we will demonstrate its simplicity through a practical, step-by-step example. First, we need an object database. Well, database sounds a bit hard, we could just make a folder for it instead in our git repository. $ mkdir .git/objects </code></pre> Now, we also need a collection of refs. No issue, just make a folder for them. $ mkdir -p .git/refs/heads </code></pre> What are heads? Don't worry, we'll get to it. Now we have the bare minimum for a repository thou, right? Well, let's try to find out! $ git status fatal: not a git repository (or any of the parent directories): .git </code></pre> It appears we are missing a critical component. $ echo "ref: refs/" > .git/HEAD </code></pre> Now, is this finally a repository? Let's see what git thinks the status is. $ git status On branch main No commits yet nothing to commit (create/copy files and use "git add" to track) </code></pre> Seems like it is! …So now we have a git repository, that was easy. Committing to a Repository</h1> Creating Objects</h2> While creating a git repository manually is an interesting exercise, it is of little value unless we can populate it with files. We likely will want to fill this repository with text files. For the occasion, you can use fortune to generate something, but if you use what I got, you'll be able to compare your hashes to the ones in the article. "If a listener nods his head when you're explaining your program, wake him up. </blockquote> That is very wise indeed. We want to add this to the repository. But how? Well, one way to add it is the following, using the git hash-object</code> subcommand. $ echo "If a listener nods his head when you're explaining your program, wake him up." | git hash-object --stdin -w 665e95f1674e9466cb429bdfebaf1b8792ef0eec </code></pre> Okay, two questions now. What is 665e95f1674e9466cb429bdfebaf1b8792ef0eec</code>, and what did hash-object</code> just do? We can inspect the git repository to investigate. Examining the directory tree structure provides some insight. $ tree .git .git/ ├── HEAD ├── objects │ └── 66 │ └── 5e95f1674e9466cb429bdfebaf1b8792ef0eec └── refs └── heads 4 directories, 2 files </code></pre> Here, we see that our object “database” in the objetcs folder has seen some change. There is a folder, 66</code>, with a file that has an equally vexing name, 5e95f1674e9466cb429bdfebaf1b8792ef0eec</code>. I wonder what is in this file. Let's try to take a look at it. $ cat .git/objects/66/5e95f1674e9466cb429bdfebaf1b8792ef0eec WKX Y$b `݆!rB-8}ɢ,xSOŋE598}y? % </code></pre> The file does not contain plain text; it appears to be in a binary format. We can use the file</code> command to determine its type. $ file .git/objects/66/5e95f1674e9466cb429bdfebaf1b8792ef0eec .git/objects/66/5e95f1674e9466cb429bdfebaf1b8792ef0eec: zlib compressed data </code></pre> Apparently, it's some data compressed with zlib. That means it's likely DEFLATE compressed (RFC 1951) in a zlib wrapper (RFC 1950)^{[2]</a>.} There is actually a neat hack we can use to view this data[3]</a>. $ printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" | cat - .git/objects/66/5e95f1674e9466cb429bdfebaf1b8792ef0eec | gzip -dc blob 78If a listener nods his head when you're explaining your program, wake him up. gzip: stdin: unexpected end of file </code></pre> We have located our string. The output begins with blob</code>, which specifies the git object’s file type, followed by 78</code>, indicating the blob’s size in bytes[4]</a>. Objects in git have 3 primary types: blob</li> tree</li> commit</li> </ul> When we ran the hash-object</code> subcommand, we created a blob. Regarding the name, that is just the sha1 hash (RFC 3174) of the object, and some other metadata and such[5]</a>. The actual output of the command was this hash. It is 40 characters long. The name of the directory in the object database is the first two characters of the hash, that is 66</code>, and the actual object files name is the 38 other characters. Manually decompressing objects for inspection is inefficient. Instead, we can use the hash 665e95f1674e9466cb429bdfebaf1b8792ef0eec</code> to inspect the newly created object. Inspecting Objects</h2> We do this with the git cat-file</code> subcommand. To see the file type of an object, we use the type flag -t</code>. $ git cat-file -t 665e95f1674e9466cb429bdfebaf1b8792ef0eec blob </code></pre> That means that the type (-t</code>) of the object we made is a blob, as we saw by inspecting it. What if we wanna see the contents of the blob? We use -p</code> for print. $ git cat-file -p 665e95f1674e9466cb429bdfebaf1b8792ef0eec If a listener nods his head when you're explaining your program, wake him up. </code></pre> This procedure is significantly more convenient than the manual decompression hack used earlier. Also, another useful flag to know is -s</code>, for size. $ git cat-file -s 665e95f1674e9466cb429bdfebaf1b8792ef0eec 78 </code></pre> This matches the 78</code> observed earlier, confirming the object size in bytes. Index</h2> Okay, enough about the blob. It seems like we have added an object – with some text – into the git repository now. We might wonder if this is reflected in the git status</code> of the repo. $ git status On branch main No commits yet nothing to commit (create/copy files and use "git add" to track) </code></pre> Adding an object to the database stores the data, but it does not tell git where that file belongs in your project. This is the purpose of the index (or staging area): to build the tree structure for the next commit. Without being registered in the index, our blob is essentially a “loose” object. If it remains unreferenced, it will eventually be removed by git gc</code> (garbage collection). To actually commit our object, we must first register it in the index. git update-index --add --cacheinfo 100633 665e95f1674e9466cb429bdfebaf1b8792ef0eec truth.txt </code></pre> The long string is the hash of the blob object we created, and truth.txt</code> is a fitting filename. What about 100633</code>? Looking at man git update-index</code> we see that… --cacheinfo <mode>,<object>,<path>, --cacheinfo <mode> <object> <path> Directly insert the specified info into the index. For backward compatibility, you can also give these three arguments as three separate parameters, but new users are encouraged to use a single-parameter form. </code></pre> Although the documentation suggests the single-parameter form for new users, we have used the separate parameter form here for clarity. The value 100633</code> is the mode of the file. In git, this mode is a 6-digit octal number that defines the file type and permissions, similar to standard UNIX file modes but restricted to a few specific values. But crucially, for blobs in git, we only have 3 modes available^{[6]</a>:} 100644 a normal file</li> 100755 a executable file</li> 120000 a symbolic link</li> </ul> Let's take a look at the repo structure again. $ tree .git .git/ ├── HEAD ├── index ├── objects │ └── 66 │ └── 5e95f1674e9466cb429bdfebaf1b8792ef0eec └── refs └── heads 4 directories, 3 files </code></pre> Here we notice that there now is an index</code> file. So all we did was add some index file</code>? …well, slow down. Let's do a sanity check with git status</code>. $ git status On branch main No commits yet Changes to be committed: (use "git rm --cached <file>..." to unstage) new file: truth.txt Changes not staged for commit: (use "git add/rm <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) deleted: truth.txt </code></pre> Aha! So we did something, we staged (added) the truth.txt file… but we also have unstaged changes saying we deleted it? Well, yes. We have told git that the blob object we created is in the working-tree, and it has the actual blob for this in its object database… but apparently can't find the file we told it about. This makes sense, since we just created the blob from stdin and didn't actually add any files. Because the file exists in the index but is missing from the actual working directory, git correctly assumes it has been removed. Hmm, while we are at it, let's take a look at .git/index</code>, just to see what that is about. $ cat .git/index DIRCf^gNfB truth.txtqblr33il% </code></pre> This confirms it is not a text file. We can verify its type using the file</code> command. $ file .git/index .git/index: Git index, version 2, 1 entries </code></pre> The output identifies it as a Git index, version 2</code> with 1 entries</code>. This single entry corresponds to the file we just added. To see what this actually looks like on disk, we can inspect the raw binary data. $ hexdump -C .git/index 44 49 52 43 00 00 00 02 00 00 00 01 00 00 00 00 |DIRC............| 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00 00 00 00 00 00 81 a4 00 00 00 00 00 00 00 00 |................| 00 00 00 00 66 5e 95 f1 67 4e 94 66 cb 42 9b df |....f^..gN.f.B..| eb af 1b 87 92 ef 0e ec 00 09 74 72 75 74 68 2e |..........truth.| 74 78 74 00 c1 b9 71 62 b4 ab c1 c2 6c 72 33 33 |txt...qb....lr33| 92 69 92 99 ac 6c 8e ff |.i...l..| </code></pre> (I removed the offset from the output) The first 4 bytes, DIRC</code>, are the “magic number” — a constant value used to identify the file format (short for “Directory Cache”). Following some padding bytes[7]</a>, the 02</code> indicates the version number. The rest of the file contains the entry data, including the filename truth.txt</code>. If we were to add another file, likely patterns would emerge in the binary structure. Investigating how the index looks in a larger project is an excellent way to learn more, so I have included it in the Exercises section below. Tree Objects</h2> Back on track, we've now made the index thingy, and we'll have to write it with the write-tree</code> subcommand. $ git write-tree a6325f064bac723691f20c0b1ed2bea82a1728fd </code></pre> git status</code> does not seem to have changed, but if we check what's in repo's file structure, we'll see something did change. $ tree .git .git ├── HEAD ├── index ├── objects │ ├── 66 │ │ └── 5e95f1674e9466cb429bdfebaf1b8792ef0eec │ └── a6 │ └── 325f064bac723691f20c0b1ed2bea82a1728fd └── refs └── heads 5 directories, 4 files </code></pre> We notice the a6325f064bac723691f20c0b1ed2bea82a1728fd</code> sha-1 hash refers to a new object. Interesting, wonder what this is. To thoroughly understand this object, we will inspect it manually by examining the raw file. This is the “hard way,” but it offers the most insight into the underlying data structure. $ file .git/objects/a6/325f064bac723691f20c0b1ed2bea82a1728fd .git/objects/a6/325f064bac723691f20c0b1ed2bea82a1728fd: zlib compressed data </code></pre> Okay, that's zlib compressed data. We know that, so let's use that same hack to figure out what is inside of this blob. $ printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" | cat - .git/objects/a6/325f064bac723691f20c0b1ed2bea82a1728fd | gzip -dc tree 37100644 truth.txtf^gNfB gzip: stdin: unexpected end of file </code></pre> This object is not a blob, but a tree — one of the three primary git object types. The number 37100644</code> initially appears confusing, but it is actually two separate values concatenated: the size (37</code> bytes) and the file mode (100644</code>). Following this are the filename and a binary sequence f^gNfB</code>. The sequence f^gNfB</code> matches data we previously observed in the hexdump</code> of the .git/index</code> file. Identifying its specific purpose is left as an exercise for the reader. Let's look at the contents the canonical way. $ git cat-file -p a6325f064bac723691f20c0b1ed2bea82a1728fd 100644 blob 665e95f1674e9466cb429bdfebaf1b8792ef0eec truth.txt </code></pre> An Experiment with Tree Objects</h2> An important concept to explore is how a tree object handles multiple files. To demonstrate this, we will create another blob. $ echo "AMOGUS" | git hash-object --stdin -w f58617716d903fb842b5606a335ff1406b9a21d3 </code></pre> And add it to our index file. $ git update-index --add --cacheinfo 100633 f58617716d903fb842b5606a335ff1406b9a21d3 amogus.txt </code></pre> Now, let's look at the repository. We have 3 objects in our objects database. $ tree .git .git ├── HEAD ├── index ├── objects │ ├── 66 │ │ └── 5e95f1674e9466cb429bdfebaf1b8792ef0eec │ ├── a6 │ │ └── 325f064bac723691f20c0b1ed2bea82a1728fd │ └── f5 │ └── 8617716d903fb842b5606a335ff1406b9a21d3 └── refs └── heads 6 directories, 5 files </code></pre> Let's dump the index binary. $ hexdump -C .git/index 44 49 52 43 00 00 00 02 00 00 00 02 00 00 00 00 |DIRC............| 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00 00 00 00 00 00 81 a4 00 00 00 00 00 00 00 00 |................| 00 00 00 00 f5 86 17 71 6d 90 3f b8 42 b5 60 6a |.......qm.?.B.`j| 33 5f f1 40 6b 9a 21 d3 00 0a 61 6d 6f 67 75 73 |3_.@k.!...amogus| 2e 74 78 74 00 00 00 00 00 00 00 00 00 00 00 00 |.txt............| 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00 00 00 00 00 00 81 a4 00 00 00 00 00 00 00 00 |................| 00 00 00 00 66 5e 95 f1 67 4e 94 66 cb 42 9b df |....f^..gN.f.B..| eb af 1b 87 92 ef 0e ec 00 09 74 72 75 74 68 2e |..........truth.| 74 78 74 00 54 52 45 45 00 00 00 06 00 2d 31 20 |txt.TREE.....-1 | 30 0a 94 c6 2b 91 4b 84 9d 9a 2e 3c 20 e4 3e 93 |0...+.K....< .>.| 1b 69 3f 19 3d bd |.i?.=.| </code></pre> Nothing too spectacular here, we notice that we see both amogus.txt and truth.txt. We also see TREE at the end of the file, that must be our tree object. However, what happens when we run write-tree</code>? $ git write-tree aee76412ed220742aeaf02ca1c50519bcea013e1 </code></pre> Let's dump the index again. $ hexdump -C .git/index 44 49 52 43 00 00 00 02 00 00 00 02 00 00 00 00 |DIRC............| 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00 00 00 00 00 00 81 a4 00 00 00 00 00 00 00 00 |................| 00 00 00 00 f5 86 17 71 6d 90 3f b8 42 b5 60 6a |.......qm.?.B.`j| 33 5f f1 40 6b 9a 21 d3 00 0a 61 6d 6f 67 75 73 |3_.@k.!...amogus| 2e 74 78 74 00 00 00 00 00 00 00 00 00 00 00 00 |.txt............| 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00 00 00 00 00 00 81 a4 00 00 00 00 00 00 00 00 |................| 00 00 00 00 66 5e 95 f1 67 4e 94 66 cb 42 9b df |....f^..gN.f.B..| eb af 1b 87 92 ef 0e ec 00 09 74 72 75 74 68 2e |..........truth.| 74 78 74 00 54 52 45 45 00 00 00 19 00 32 20 30 |txt.TREE.....2 0| 0a ae e7 64 12 ed 22 07 42 ae af 02 ca 1c 50 51 |...d..".B.....PQ| 9b ce a0 13 e1 07 2b 99 c5 b6 0e 3d 53 33 c9 21 |......+....=S3.!| dd b1 75 41 41 84 b1 d8 ec |..uAA....| </code></pre> The data following the TREE</code> identifier has increased in size. We will now inspect the contents of this newly written tree. $ git cat-file -p aee76412ed220742aeaf02ca1c50519bcea013e1 100644 blob f58617716d903fb842b5606a335ff1406b9a21d3 amogus.txt 100644 blob 665e95f1674e9466cb429bdfebaf1b8792ef0eec truth.txt </code></pre> Now we seem to have two files inside of the tree. Time to Commit</h2> So now, it's time to create a commit. Above, when we did write tree, we created the tree</code> object aee76412ed220742aeaf02ca1c50519bcea013e1</code>. This includes both of the blobs we made. So how do we commit that? It's actually really simple, we just use git commit-tree</code>. git commit-tree aee76412ed220742aeaf02ca1c50519bcea013e1 -m "initial commit" 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6 </code></pre> If we now look at the .git</code> repository, we will see a new object, 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6</code> created. We can verify the type of this new object. git cat-file -t 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6 commit </code></pre> Unsurprisingly, it's a commit object. If we look what is inside, we see something that may look familiar to people that regularly use git. git cat-file -p 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6 tree aee76412ed220742aeaf02ca1c50519bcea013e1 author Christina Sørensen <christina@cafkafk.com> 1715792150 +0200 committer Christina Sørensen <christina@cafkafk.com> 1715792150 +0200 initial commit </code></pre> A fun thing to do now is to try and run git log</code>. $ git log fatal: your current branch appears to be broken </code></pre> Seems like our branch is broken hu? Let's fix that. Let's quickly remind ourself of the contents of .git/HEAD</code>. $ cat .git/HEAD ref: refs/ </code></pre> This doesn't refer to anything. We can easily fix this, let's make a new branch. But what git subcommand will we use this time? None actually, as it turns out, branches are just files in .git/refs/heads/</code> that contain the sha-1 hash of some commit. The name of the file becomes the name of the branch. $ echo 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6 > .git/refs/heads/main </code></pre> Now we just need to switch to that branch. Instead of doing git switch main</code> we can just change the reference in the repository directory .git</code>. $ echo "ref: refs/heads/main" > .git/HEAD </code></pre> If you're following along, and you have some PS1 git branch feature, you may have noticed something incredible just after running that command. First, if we run git status</code> now, we see that there no longer are any changes to be committed. $ git status On branch main Changes not staged for commit: (use "git add/rm <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) deleted: amogus.txt deleted: truth.txt no changes added to commit (use "git add" and/or "git commit -a") </code></pre> Also, we can now run git log</code>! git log commit 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6 (HEAD -> main) Author: Christina Sørensen <christina@cafkafk.com> Date: Wed May 15 18:55:50 2024 +0200 initial commit </code></pre> Something else that's cool is we can run git log --format=raw</code>, and see output similar to what we got from git cat-file -p</code> on the commit objects sha-1. $ git log --format=raw commit 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6 tree aee76412ed220742aeaf02ca1c50519bcea013e1 author Christina Sørensen <christina@cafkafk.com> 1715792150 +0200 committer Christina Sørensen <christina@cafkafk.com> 1715792150 +0200 initial commit $ git cat-file -p 87a1aa833dccca5ea503e9a7ff81c51fe82c85c6 tree aee76412ed220742aeaf02ca1c50519bcea013e1 author Christina Sørensen <christina@cafkafk.com> 1715792150 +0200 committer Christina Sørensen <christina@cafkafk.com> 1715792150 +0200 initial commit </code></pre> But what about those files that are deleted? We can solve that with git checkout</code> like this. $ git checkout HEAD -- amogus.txt truth.txt </code></pre> Now if we run git status</code> we get. $ git status On branch main nothing to commit, working tree clean </code></pre> With this, we have successfully created a commit from scratch. Conclusions</h1> Starting from an empty directory, we have successfully initialized a repository, staged files, created a branch, and committed our work — all without using the standard git commands. Is this efficient? Certainly not. However, by bypassing the porcelain commands, we have exposed the elegant simplicity of git’s plumbing. Understanding how objects, trees, and references interact transforms the system from a black box into a logical machine. This deeper understanding will help you internalize the tool, making complex operations intuitive rather than terrifying. There are still open questions: how does a pull</code> work? How do we manually rebase</code>? That is a topic for another day — or perhaps a therapy session. For now, we will leave them as an exercise for the curious reader. Exercises</h1> What does the hexdump -C</code> look like of the .git/index</code> if we add another blob with hash-object</code> and update-index</code>?</li> Can you discern any patterns from this. What more can you learn about the index file format.</li> What does the zlib compressed data look like inside of the tree object after if we add another blob?</li> What does the .git/index</code> look like after running update-index</code> with a tree blob in the object database?</li> What does the .git/index</code> look like after running update-index</code> and write-tree</code> with a tree blob in the object database?</li> Can you figure out what the myserious f^gNfB</code> means?</li> What does the hexdump -C</code> look like of the .git/index</code> inside a larger project.</li> Can you discern any patterns from this. What more can you learn about the index file format.</li> Does changing the size of the blob from 78 to some other number inside of the zlib compressed data influence git cat-file -s</code>? You will need to decompress and recompress the data from and into the proper zlib format^{[8]</a>.</li> </ul> Footnotes</h1> We don't have to put our git repo in .git</code>. We could use the GIT_DIR</code> environment variable, or the --git-dir=<path></code> flag. ↩</a> </li> the zlib wrapper (RFC 1950) — unlike gzip wrapper (RFC 1952) — doesn't store file name and other file system information, which is fine, considering how git manages this elsewhere. ↩</a> </li> From this unix.stachexchange answer</a>. Here, we concatenate the gzip magic number and compression method, and concatenate (the actual reason for cat existing) this with the file. We then pipe it into gzip, who can now understand and decompress it. Still, we didn't finish the file with the 8 byte footer, so gzip gets confused, but that doesn't matter, we get to see the data regardless. ↩</a> </li> https://git-scm.com/book/en/v2/Git-Internals-Git-Objects</a> ↩</a> </li> If you're interested in finding out how the hash is generated, start here</a>. ↩</a> </li> Read more herehttps://git-scm.com/book/sv/v2/Git-Internals-Git-Objects ↩</a> </li> The mortal enemy of C has many names: empty bytes, null bytes, nop bytes. ↩</a> </li> One approach could be to use the hack, with the additional padding at the end, to extract the file. Then, after changing the number, compressing it again, and removing the prepend and appended gzip magic numbers. ↩</a> </li> </ol>} Institutional Math Christina Sørensen — Fri, 10 Nov 2023 14:03:00 +0000 Foreword from the future</h3> As 2025 is coming to an end… I’ve been reading over some of my old posts. The tone here is obviously on the more polemic side, but it would be a disservice to my former self to erase the record of my views on academia when I was on the inside. I think it’s worthwhile to conserve not just the more reasoned and polished version of “why did you leave academia” — that I now often find myself regurgitating — but also conserving the “emotion” behind that decision. Because any after the fact rationalization aside, the fundamental decision wasn’t rational, it was emotional. Experienced. Not up for argument. It was an incommensurable experience of a normal science that wasn’t suited to my taste. It was the observation of the fundamental sin in the heart of academia as an institution. If nothing else, then as a warning to my future self to never make the fault of Epimetheus again. Before Newtons birthday seems an apt time to spit on the grave of the enlightenment. So /spit</code>, cheers, and onward! I’ve been wanting to write something terse, dense, with academic heft on this subject a long time. Sadly, I fear I’ll never write such a thing. But I need to speak my heart. I left the mathematical asylum, but I need some closure, and eating Curry isn’t cutting it anymore. Grigori Perelman rejected the fields medal, and the Clay Institute awarded monetary prize for solving one of the “Millenium Problems”. Evariste Galois laid dormant for 10 years after his death in the drawers of Liouville. Gauss and Newton let their personal quibbles turn into abuse of power. Grothendieck left the whole thing behind, burned all his papers. Kaczynski became a domestic terrorist. Lovely field indeed! I’m not interested in money or fame; I don’t want to be on display like an animal in a zoo. – Grigori Perelman</cite> </blockquote> The QED project is but ruins — visions of better times. If the proof isn’t surveyable, then it isn’t solvable! Mathematics isn’t just syntactics, no, they’re new age incarnations on the platonic plane of ideas, they’re universal truths of the universe!… Nonsense. Mathematics is circlejerking on tautological systems. Lockhart can lament all he wants, at least in primary school the problems are trivial, and if you’re done fast you can do something actually useful with the research stipend called public school. At the prestigious academy, the queen of sciences is growing ever more complicated year by year… but ironically, the material taught is the perfect problems for proof systems, we’re teaching students only to replicate. Since the primary concern of these problems is less work for the validator, the true test of your “learning outcome” is your ability to solve solved problems systematically. We must preserve the human validator, by turning said human into a mistaken mockery of the computer. Science, knowledge, the university itself. Open is its doors to any and all. Come research for Boeing and Lockheed, at the faculty of partitioning of 7 hole toruses by means of ballistics, or if it feels too theoretical to indirectly maim and kill people in foreign countries, why not study applied mathematics, and develop social models for companies like Palantir? Profiling, what’s that? Do you not believe in the infallible probabilities? “We need a new Bourbaki” we young people may say to each other, but we all know well that we’d rather get ours and leave this corpse rotting and dying. If anything, what’s needed isn’t more fuel on the fire. Institutional science is systematically pissing in the super soaker wielded by various tyrants, profiteers and oligarchies around the planar graph we all live on. If we are to change mathematics, we should get to the roots of the problem, like we have always done. The foundation on which we built has a crisis, and that foundation isn’t mathematical, it’s social. And for myself, for my mathematical circlejerking? I keep it between friends, private, like good sex. The smut we produce we encrypt and keep safe so as to not show something profane to the plutocrats of the academy. Encrypting — hoping to be kept in those drawers until saner heads emerge to open it. RIME Christina Sørensen — Thu, 09 Nov 2023 16:02:00 +0000 This isn’t a tutorial, it’s more a “here’s what that thing I’ve been working on is about.”[1]</a> </blockquote> I like Nix a lot. It’s a so called “obvious good idea”. Even with much of its baggage, some technical, some cultural, it’s probably one of the most interesting things currently happening in software, and you’re making a mistake if you’re not paying attention to it. My desktop, laptop, build server, and even my IRC-bounce server in the “cloud” runs Nix. This makes it easy for me to provision my systems from a single source of version controlled truth. What has made that even better is that all the software projects I maintain include a so called Nix flake, and I make sure to upstream the projects I could imagine people would find some value in to nixpkgs every week on release. One particular project I maintain is called eza, it’s that fork of exa, that you may or may not even be aware that you’re running if you’re a former exa user[2]</a>. Eza relies heavily on Nix. The first way Nix showed up was to help us remove vagrant, a product from HashiCorp, the company famous for making terraform. This was partially because I was uncomfortable with the vagrant business license, but further because Nix could replace it while providing unique benefits. But we do so much more with Nix. The Nix flake sets up the development environment for contributors, in a way that matches the same environment as I myself develop on effortlessly for devs. It helps ensure that commits follow conventional commits, and checks if the code conforms to a battery of linters. It further makes it easy to format all source, and helps build eza in a declarative way. In general, I like to think that we at eza run a tight ship. This is necessary, because we’re working in a surprisingly hard domain, where there is a lot of side-effects — what else would you expect from something so closely tied to the file system. This means that we enforce semantic versioning as well, and generate changelogs that make breaking changes clear. Now with all that in mind, let me quickly explain the lifecycle of a eza version. We start on Thursday, after a release was just made. For most of the week, we slowly and steadily merge pull requests. Then, when Thursday comes, based on whether or not a breaking change was introduced (as is easily parsed from our commit summaries thanks to conventional commits), we determine the next version of eza automatically, and justfile recipes (think make files but less pain) cross compiles binaries for the platforms we support, and cuts and tags the new release, as well as making a “proper” github release. Then, after all that is done, there’s only two things left. First, I run cargo publish</code>, to send the newest eza version to crates.io, from which it is immediately available for download via cargo install eza</code>[3]</a>. Secondly, I update the nixpkgs package (that I, reasonably enough, am the maintainer of), and await someone with commit bit set to merge it. Now, it is often merged quite quickly these days, I think that there is a small group of commiters who’s gotten used to looking for my batch of software updates and merge them without much fuss. But even with a same day merge, due to the nature of nixpkgs, this newest version will actually first be available in some days, usually four to five. This is because the nixpkgs unstable branch lags behind the main branch[4]</a> by a bit. The reasons for why this is are overly technical, but sufficient to say that even while nixpkgs-unstable is the most up to date package distribution in aggregate, compared to arch-extra, which usually has the newest version of eza within a day, for packages like eza, it lags behind. There of course is a potential solution to this, one that I did use for bit. See, since eza has a flake.nix</code> file, one can just refer to that directly in their NixOS config, by making it a flake input (if one does use flakes that is). This is great, that way, you don’t have to wait for nixpkgs-unstable to catch up with our latest release, but there is a catch. Flake inputs have no notion of versioning. Now, if you’ve been keeping your ear to the ground, there has, withing the last few month been two major improvements on this front. Firstly came flakehub, a proprietary platform by many NixOS veterans. Later came the, seemingly spiteful flakestry.dev, by other NixOS veteran. Between the lines, it’s farily clear that this is a direct reaction to flakehub being closed source. The selling point of both of these platforms is the following: you add a github action, and whenever you make a proper github release, that action will push your flake to the respective platform. The problem with both however, at least to me, as someone that distributes software to a fair few amount of users, but even further, just as someone that likes to tinker with my system and loves to make it all neat is the following: People need to add a github action to their repos.</li> Only repos that have done this, and has released after, can actually be used.</li> This only works for github.</li> If this brittle setup breaks in the future, of if releases stop coming, you’ll have to move to a different source.</li> </ul> And I tried to see if these limitations weren’t too annoying for me to use for some months. But aside from the technical limitations, the closed nature of flakehub got to me (at the time, flakestry.dev didn’t exist). Having had this in the back of my head lead to many shower thoughts, which lead to additional shower thoughts, which lead to generally very long showers. Some weeks ago, those shower thoughts collided into a sudden, immediate hacking run, and rime was born. Now, three weeks later, we’re sorta at a place where all the essential features we want are in place. Those things include supporting not just github, but gitlab, codeberg, forgejo, gitea, sourcehut, and even flakehub[5]</a>, supporting semver requirements, whether that be getting any version that is a version $\le 0.3.48$, or just only getting minor and patch updates, but not major version bumps without manual intervention. And like, a lot more features. But really, what makes it great is that I can just specify: { description = "My Configuration"; inputs = { eza.url = "http://rime.cx/v1/github/cafkafk/eza.tar.gz"; ... your other inputs } outputs = { ... all your cool outputs } } </code></pre> And like that, the eza flake input will always be the latest released (excluding pre-releases) version. This really puts nixpkgs into perspective for me. For projects that already have their own flakes, the value of nixpkgs mainly becomes it being a hand crafted manually updated package whenever there is a new version. And while I’m determined to make eza available to as many users as possible, and having it in nixpkgs will obviously help that… for my own use at least, it’s just so much more neat to have it as a flake input, but without having to include unreleased, potentially unstable changes. It clearly to me also opens some routes to a more decentralized NixOS landscape, since it adds a sort of layer between the latest raw flake and the users computer, reminding me of the layer of snow that build up when you place something sturdy in a hostile, cold climate with a lot of snow(flakes). A layer called rime. Although much of the work has been done by Gergely Nagy</a>, who I’ve been able to successfully nerdsnipe. ↩</a> </li> Since distributions shim in eza in the place of exa nowadays. ↩</a> </li> Although, sadly, it’s impossible to include man pages and completions via crates.io. ↩</a> </li> Technically, the “master” branch, but let’s not indulge that anachronism. ↩</a> </li> And it’s written to be easily extendible, so adding new forges is trivial. ↩</a> </li> </ol> Adding dollar sign (apply) to nix Christina Sørensen — Thu, 03 Aug 2023 14:34:33 +0000 This post is very low quality for comedic effect. </blockquote> A detailed list of instructions</h2> Take one nix</a>, apply this patch</a>[1]</a>: What does this do? It adds a new token $</code> to the nix languages parser and exposes it as __dollar</code>, essentially by copying the way this works for __lessThan</code>. Hack on your nix: nix develop ./bootstrap.sh ./configure $configureFlags --prefix=$(pwd)/outputs/out make -j $NIX_BUILD_CORES outputs/out/bin/nix repl </code></pre> Now in the nix repl</code> that just opened: nix-repl> __dollar = x: y: builtins.foldl' (a: b: a b) x y </code></pre> This makes $</code> an infix operator that’s essentially just a generic apply. Now just create a toy example to demonstrate: nix-repl> concat = x: y: z: (x + y + z) nix-repl> concat $ ["literally" " " "dollar sign"] </code></pre> And your repl will say: "literally dollar sign" </code></pre> Tadaa! That’s all it takes to add the $</code> operator to nix, just like Haskell. The end $\text{\(^ヮ^)/}$ diff --git a/src/libexpr/parser.y b/src/libexpr/parser.y index 201370b90..4b26af623 100644 --- a/src/libexpr/parser.y +++ b/src/libexpr/parser.y @@ -353,6 +353,7 @@ void yyerror(YYLTYPE * loc, yyscan_t scanner, ParseData * data, const char * err %left AND %nonassoc EQ NEQ %nonassoc '<' '>' LEQ GEQ +%nonassoc '$' DOL %right UPDATE %left NOT %left '+' '-' @@ -411,6 +412,8 @@ expr_op | expr_op LEQ expr_op { $$ = new ExprOpNot(new ExprCall(makeCurPos(@2, data), new ExprVar(data->symbols.create("__lessThan")), {$3, $1})); } | expr_op '>' expr_op { $$ = new ExprCall(makeCurPos(@2, data), new ExprVar(data->symbols.create("__lessThan")), {$3, $1}); } | expr_op GEQ expr_op { $$ = new ExprOpNot(new ExprCall(makeCurPos(@2, data), new ExprVar(data->symbols.create("__lessThan")), {$1, $3})); } + | expr_op '$' expr_op { $$ = new ExprCall(makeCurPos(@2, data), new ExprVar(data->symbols.create("__dollar")), {$1, $3}); } + | expr_op DOL expr_op { $$ = new ExprOpNot(new ExprCall(makeCurPos(@2, data), new ExprVar(data->symbols.create("__dollar")), {$3, $1})); } | expr_op AND expr_op { $$ = new ExprOpAnd(makeCurPos(@2, data), $1, $3); } | expr_op OR expr_op { $$ = new ExprOpOr(makeCurPos(@2, data), $1, $3); } | expr_op IMPL expr_op { $$ = new ExprOpImpl(makeCurPos(@2, data), $1, $3); } diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index ddf529b9e..00622a142 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -3583,6 +3583,24 @@ static RegisterPrimOp primop_lessThan({ .fun = prim_lessThan, }); +static void prim_dollar(EvalState & state, const PosIdx pos, Value * * args, Value & v) +{ + state.forceValue(*args[0], pos); + state.forceValue(*args[1], pos); + // pos is exact here, no need for a message. + CompareValues comp(state, noPos, ""); + v.mkBool(comp(args[0], args[1])); +} + +static RegisterPrimOp primop_dollar({ + .name = "__dollar", + .args = {"e1", "e2"}, + .doc = R"( + Just like haskell :) + )", + .fun = prim_dollar, +}); + /************************************************************* * String manipulation </code></pre> ↩</a></li> </ol> The Pull-Request Heist Christina Sørensen — Thu, 03 Aug 2023 14:34:33 +0000 Your effort to remain what you are is what limits you. – The Puppet Master</cite> </blockquote> The Beginning</h2> May 3, 2014. Benjamin “Ogham” Sago has just made his first commit to Exa</a>. This will go on to be one of the most used ls alternatives, with ~22k stars, 655 forks, and 151 contributors. For almost a decade, he maintains exa. Yet, around 2021, activity is drastically slowing. Eventually, the project falls dead. With 63 open pull requests, 276 issues, and some major bugs, the project is on its deathbed. The users are begging</a> for activity. At this point, other projects are being advertised in the issues for various features. Features that — mind you — are already finished in the pull-requests, but without anyone to merge. The situation is dire, are we all doomed to use LSDeluxe</a> and endure its awful color scheme?[1]</a> That’s where I somehow came in. The rabbit hole yak-shave</h2> The terminal emulator kitty is really neat. So is exa, and I’ve been using exa for a fairly long time. Never have I given it much more thought than installing it from nixpkgs, I’m for the most part blisfully unaware of the projects state (which speaks to it’s quality no doubt). Kitty has a feature. You can type ctrl+shift+e</code>, and it highlights links with letters, a bit like vim easy-motions, and allows you to effortlessly open them in your browser. As a keyboard only extremist, this pleases me. Of course, the thing is, when I exa some directory, I can’t “easy motion” for files. And I want that. So I look into it. Turns out, kitty has that feature! And so does ls! ls --hyperlink=auto </code></pre> Neat! Then, tragedy… exa doesn’t. What! A bit of searching reaveals to me a PR on exa. It’s been there since April. It’s awsome! But it isn’t merged. Why? I orient myself in the issues. I try to figure out who to complain to. But there is none, the project looks dead. I dip my toes in some forks, see if I can get this done. Seems trivial. Then, my partners has to help sister move. I have the weekend free. The nerd snipe tinglies are coming on. A hacking run</a> is imminent — whether I like it or not. Onto the programming</h2> Fork other forks, we’ll write our own! Call it eza, old habits die hard, and the x is next to the z on my keyboard. This will suffice. In one bold move we have already solved half of the hard problems of computer science[2]</a>, I’d say that’s a good start. Now we just have to get those pull request over in our repo. One way to do this is the following: git fetch \ git@github.com:ogham/exa.git/pull/<pr-id>/head:<target-branch> </code></pre> This way, we can take a pull request branch from exa, and copy it into a branch in our repo. The problem is, there’s 63 pull-requests!?! We’d have to do this for each of them, push the branch to github, then slowly click through the webui and make that pull request, one at a time. As a keyboard only extremist, this displeases me. There is no way we’re doing this. If only we had a list of all those repo-ids, we’d be able to at least automate the fetching. Then, lightbulb. Github has a REST-API</a> ripe for our queries[3]</a>. A bearer token is created. An absolute unit of a curl request is crafted. The world pauses for a moment as we lay the fate of the project into the wire. curl -L \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer <the-bearer-token>" \ -H "X-GitHub-Api-Version: 2022-11-28" \ 'https://api.github.com/repos/ogham/exa/pulls?per_page=100' > open-prs-full.json </code></pre> Then we wait for the response. After the Suspense</h2> Well. It worked. cat open-prs-full.json | head -n 10 </code></pre> [ { "url": "https://api.github.com/repos/ogham/exa/pulls/1221", "id": 1446083658, "node_id": "PR_kwDOATIZtM5WMXxK", "html_url": "https://github.com/ogham/exa/pull/1221", "diff_url": "https://github.com/ogham/exa/pull/1221.diff", "patch_url": "https://github.com/ogham/exa/pull/1221.patch", "issue_url": "https://api.github.com/repos/ogham/exa/issues/1221", "number": 1221, ... </code></pre> Was this what we needed? We do see a number</code> property there, and if we look in the html_url</code> property, we see that it matches the expected url for that pull request. Now here is the part where I’d shown off my jq prowess, but I have no such thing. So instead, I make do with some regex. cat open-prs-full.json \ | rg -i 'pulls/\d+"' -o \ | tr -d "pulls/" \ | uniq \ | tee pr-ids.txt \ | wc -l </code></pre> 63 </code></pre> The number of lines is right, but is the output? cat pr-ids | head -n 5 </code></pre> 1221" 1219" 1210" 1204" 1202" </code></pre> Ahh, close enough, a little adjustment and we are there: cat open-prs-full.json \ | rg -i 'pulls/\d+"' -o \ | tr -d "pulls/" \ | tr -d '"' \ > pr-ids.txt </code></pre> Now did we get it? cat pr-ids.txt | head -n 5 </code></pre> 1221 1221 1219 1219 1210 </code></pre> Seems we did! The Part Where We Jam Those Pull-Requests into GitHub</h2> At this point, the estrofem tablet under my tongue had dissolved, and I’d woken from my groggy morning state. A purely performative crack of the knuckles were in order, as I leaned over my desk like a hawk that’s caught sight of a mouse. If you’re a keyboard-only extremist like me, web-ui is a nuisance to be solved. GitHub has gh</code>, or the GitHub CLI. It is extremely useful to sync your nixpkgs fork, and send your package definition pull requests. But today, old reliable was gonna learn some new tricks. Then I paused. And stared. In my head flashed visions of old Bell Labs informational videos</a> on the Unix Operating System. The voice of Brain W. Kerninghan echoed in my head — “I think the notion of pipelining is the fundamental contribution of the system…”. The hacking cgi was front and center as I said out loud “It’s a UNIX system, I know this!”. Temporarily absolved of my mortal coil, I resided in waves of abstraction and shell, time and language lost their meaning to the beauty of the baud. It was as if every bit moved in concert with my thoughts, as if the collective will of every electron in my sillicon rock filled out the sails of my spaceship gently surfing on the stream of raw information entering and exiting me. As I slowly raised my cup of tea to my mouth, not for one moment breaking this intimate emergence and embodiment of me in my code, in my will, in my craft, before me it appeared, at the end of time and space itself. The final destination, the one truth of programming, the zen was ever closer. Then the tea — hotter than a NVidia GPU running KDE — removed my taste buds faster than a Arch maintainer removes system critical boot code. What had been my ticket to enlightenment turned into a leaky abstraction all over my desk. However, I had tried so hard and gone so far, there was no way that in the end, it wouldn’t really matter. Now — armed with a more modest disposition, xargs, and the GitHub CLI — I went to work. Soon, emerged the command that would seal my fate: cat pr-ids.txt \ | xargs -I% -n1 sh -c \ 'git fetch git@github.com:ogham/exa.git pull/%/head:pr-%; git checkout pr-%; git push; gh pr create --title \'(exa PR) %\' --body \'github.com/ogham/exa/pulls/%\'' </code></pre> As a smut novel once said[4]</a>: There comes a time in every woman’s life where she reaches a crossroads. A chance to choose between two paths. One is unknown, filled with potential and struggles that demand growth and change, but could lead to new beginnings. The other is familiar, grooved with well-worn ruts and established twists, muddy holes and isolation. The path that, while lonely, fits like a glove. When — if — I hit enter, I’d be down the road of committing myself to maintaining the damn thing. Were I not to, I’d never downlive the internal shame. Were I to do it, I’d have allocated quite a substantial amount of time in my life to pesky users, stubborn contributors, and the chore and dull drag of changelogs, version bumps, and releases. As I hit enter, and as the pr tab on the repo slowly grew in the web interface, I hoped this wouldn’t be a mistake. The Aftermath</h2> Over the next two days, me and sbatial</a> would go onto hack on eza. When monday came, 63 had been reduced to 18. Hyperlinks, SE-linux contexts, git repo statuses, human readable relative dates, more security fixes than I have fingers to count them on. And the darn grid bug, the pesky bugger had been swatted and was now but a stain on the git history. With flakes, CI, and sheer will, we had overcome. Trying out eza was as imple as running: cargo install eza </code></pre> Or if one was so inclined: nix run github:cafkafk/eza </code></pre> But had we won? The battle, no doubt, but not the war. Reflections on DevOps and Infra</h2> In the excellent talk at Chaos Communication Conference 36: From Managerial Feudalism to the Revolt of the Caring Classes</a>, the late David Graeber summarizes a core problem in modernity as eloquently as only he could: [Whenever] I say, talk to a Marxist theorist, wherever, they try to explain value, which is what they always like to do, they always take the example of a [bottle]. […] Well, look at this bottle, you know. [It] takes a certain amount of socially necessary labor time to produce this, say it takes, you know, this much time, this much resources. It’s always about production of stuff. But a teacup or a bottle. […] You produce a cup once, you wash it like 10000 times. Most work isn’t actually about producing new things, it’s about maintaining things. […] So I think that what we need to do is we need to start over. We need to realign. First of all, think about [work], not as producers, but as carers. </blockquote> I spend four years of my life studying “the queen of the sciences”, mathematics herself. One view that I particularly enjoyed were that of the toolmaker. Sure, we were studying abstract structures, but that’s being a student. A mathematician was a tool maker. That view might have a stink of applied mathmatics, but I think it’s true even for those who’d call themselves pure[5]</a>. This too is true in DevOps, in systems programming, in infrastructure. At the end of the day, we’re toolmakers. In my mispend youth, in the grips of startup culture, the idea of the elusive 10x developer, while just a mirage was still very alluring. Regardless of the obviously wrong and problematic discurse related to it, there is no doubt that some people move mountains, and others don’t. I wanted to move mountains. What I’ve later came to learn is that’s just an illusion. The idea of the lone genious, burning the midnight oil is just a myth. John Carmack, Linus Torvalds, George Hotz, they’re just a personality cult. And this isn’t just some clever insight I’ve come to, it’s stolen</a>. But I like my iteration on it. A certain Archmage of Infrastructure has the right idea about being a “force multiplier”. What really matters to move these mountains is to enable those that want to move them. The most dedicated cyclist in the world will cycle less than an average bikesmith will enable others to do. In this creating of tools, and in this care work of maintaining them, we are force multipliers, and in our multiplication of force, the impossible becomes possible. This is the point of DevOps. This is the point of infra. This is the next level of abstractions hackers should strive for if they wanna boil the ocean. Post-Credit Scene</h2> From 5:00-19:00, for two days, I had hacked. 3 meals a day, hydrating, and a walk in nature had supported this endeavor[6]</a>. At last, a knock on the door, my partner had come home. After a hug and some pleasantrise, came the question: So, what did you do with your weekend? </blockquote> I eagerly typed eza</code> in my terminal emulator, and showed how there were now letters next to the entries, and clicking them copied the text. What should have been pure awe was a look of slight concern, followed by a shoulder shrug and a kiss on the forehead. That’s all you did while I was gone? </blockquote> It’s all love, I bet it’s just a bad screenshot on their git repo <3 ↩</a> </li> See the first quote here</a>. ↩</a> </li> And we’d never scrape a website, that’s rude :^) ↩</a> </li> “Arabella Stone, Nashville’s darling, is eager to shuck her prim-and-proper rep, and a few wild months spent checking items off her ’ summer bucket list’ is the way to do it. First up: kiss the man she’s crushed on since she wore a training bra, the bad boy of country himself, Blue bassist Charlie Tucker. For Charlie, a beautiful woman flirting with him isn’t out of the norm—but a beautiful woman bolting after the hottest kiss of his life sure is. And when he finds out his kiss-and-run Cinderella is none other than Arabella Stone, daughter of his label’s CEO, he knows he’s in trouble. Because not only is she a Stone, she’s also his employee for the next few months at the recording studio he just bought. Over the course of one thrilling summer, Arabella and Charlie chip away at her bucket list and fight the simmering attraction between them…knowing that once it’s all over, so is their time together.” Steamy. ↩</a> </li> Which really should be called theoretical. ↩</a> </li> All this rise and grind never sleep never eat never enjoy your life stuff is stupid. If you wanna be even like mildly consistent, you gotta get this right. Dumb stuff like hacking is like the last thing you get to in mazlows hierachy[7]</a> for a reason. ↩</a> </li> Yes, I know it’s “debunked”, it’s a working model geez louise. ↩</a> </li> </ol> Warts and all Christina Sørensen — Mon, 24 Jul 2023 16:10:10 +0000 Learning</h2> The better you get the worse you get. The more I learn the more I learn that I don’t know much. This can be overwhelming at times. I think it’s pretty normal to feel this way. But I also see it can easily become a limiting type of negative feedback. Looking back at old projects, old essays, old anything, I see flaws. I was so naive. Simultaneously, I feel like I see how my output has gotten better. But realizing this negative feedback loop — as in keeping with the theme — just makes you even more insecure! Because, if this model is true (and it generally seems to be), then this post I’m writing right now will seem so bad to me in a year… not to imagine 10 years! Knowing this, it’s easy to fall into perfectionism. There’s always something to improve after all. But this is dangerous. Because, while it might be the case that in 10 years, I can easily see all the flaws, right now I can’t. And it seems to be my experience that no matter how much I correct, no matter how much time I spend trying to make something perfect, I only realize what was actually missing much later. Worst, things don’t even make it out the door. Perfectionism can kill projects. When that happens, I don’t realize the flaws. The projects I never finished because I didn’t know how to explain it the right way or didn’t know just how to say what I wanted to say perfectly, those I still don’t know how to improve. Granted, they might just still be out of my reach, and thus simply never get finished because of this, but I feel everything I do is out of my reach, even when doing it. And then it’s done, and I hate it a little. And then a year goes by, and I hate it a lot. I have to finish things to improve. Incomplete things are in a constant state of improvement. They’re too maleable to be at fault[1]</a>. They’re beyond critique because they’re meerly in progress, they’re not solid enough to grasp and criticise. Ideally, I shouldn’t get discouraged by feeling stupid. In fact, I’m starting to feel stupid, and I’m kinda glad about that, because I know how much I had to learn to be this dumb. But I dislike everyone’s complex with wanting to be the wisest of all the G(r)eeks by pretending that they don’t know anything. Knowing that you know nothing is a paradox, and a platitude. He who knows, does not speak. He who speaks, does not know. – Lao Tsu</cite> </blockquote> This is a dumb quote. It should really be: Those who speaks, does not know. Those who knows, does not speak, and plateaus. </blockquote> You can’t know things, you can only know them more. So knowing, isn’t a destination, but a process. But it’s a painful one. Self Redaction</h2> I feel like contemporary culture expects a self redaction from the individual of their history. You’re not supposed to fail in public. In a spectacle, there are just dunkers and those that are dunked on. Being wrong is death of your socially distributed ego. What this false internet reality leaves out is the numerous failures that lie at the end of everything even remotely sort of cool. Without this ever in the public eye, the culture perpetuates and intensifies perfectionism in us all. I think that to some extend, this isn’t just a moral issue, but also an individual one. The idea of being unable to fail in public means you can’t grow in public either. This I find limiting. Further, for those around us, the network effects of being imperfect seem to be worthwhile. In fact, I’d argue that in a time of flawless, but false public personas, being not just a finished product but a evolving history is a much more potent and intersting identity to embody. And it’s one that anyone with any sense should find preferable to the superficially flawles. I feel like I’ve already made numerous public mistakes, and I don’t see that slowing down, and maybe I shouldn’t want that, each one is a sign of me going in the right direction. I don’t know how to end this. Reference implementations and their consequences have been a disaster for the human race. ↩</a> </li> </ol>

Open weights are not open source

Another NixOS Election Cycle

Tracking Sattelites with Consumer Components

Obligatory Conclusion</h2>
With inexpensive consumer hardware, it is possible to build your own sattelite ground station, and join a globe spanning network of space enthusiasts capturing sattelite data.</p>
That is pretty cool :3</p>

Conventional Commits Change my Brain

rustls and aws-sdk seems incompatible with FOSS

Merge or Rebase

Grok Git Repos

Institutional Math

RIME

Adding dollar sign (apply) to nix

The Pull-Request Heist

Warts and all

point.free

I restarted a 10 year old Xeon 174 times to delete twelve flags and gain four tokens a second

A 10 year old Xeon is all you need

Quantizing Gemma 4's MTP drafters

point.free

I restarted a 10 year old Xeon 174 times to delete twelve flags and gain four tokens a second

A 10 year old Xeon is all you need

Quantizing Gemma 4's MTP drafters

Open weights are not open source

Another NixOS Election Cycle

Tracking Sattelites with Consumer Components

Hardware</h3> If you wanna build your own, the easiest possible route is just to acquire a LoRa32 V2.1_1.6 - 433MHZ CH9102F</code> board. Usually you can find these online, and make sure that you get one with a small antenna.</p> </p>

Obligatory Conclusion</h2> With inexpensive consumer hardware, it is possible to build your own sattelite ground station, and join a globe spanning network of space enthusiasts capturing sattelite data.</p> That is pretty cool :3</p>

Conventional Commits Change my Brain

rustls and aws-sdk seems incompatible with FOSS

Merge or Rebase

Summary</h2> So what where the tradeoffs?</p>

Grok Git Repos

An Experiment with Tree Objects</h2> An important concept to explore is how a tree object handles multiple files. To demonstrate this, we will create another blob.</p>

Footnotes</h1>

Institutional Math

RIME

Adding dollar sign (apply) to nix

The Pull-Request Heist

After the Suspense</h2> Well. It worked.</p>

Warts and all

Obligatory Conclusion</h2>
With inexpensive consumer hardware, it is possible to build your own sattelite ground station, and join a globe spanning network of space enthusiasts capturing sattelite data.</p>
That is pretty cool :3</p>