============================================================== Guild: wafer.space Community Channel: Information / questions / Is magic PEX extraction expected broken for `gf180mcu_fd_sc_mcu7t5v0__fillcap_64`? After: 04/30/2026 23:59 ============================================================== [05/16/2026 03:33] namibj ``` Error in extracting node gf180mcu_fd_sc_mcu7t5v0__fillcap_64_40578/a_3260_375# Bad Device Location at 585660,629620 Couldn't find device at 585660 629620 ``` {Attachments} 2026-05_media/image-C7A06.png [05/18/2026 18:31] mithro_ @Leo Moser (mole99) / @Tim Edwards - Any ideas? [05/18/2026 18:33] namibj something wonky; It's going better now. [05/18/2026 18:34] namibj (Better as in, not failing; current ETA is 300 days though.) [05/18/2026 18:35] namibj it spits out batches of about 23 nets each, every about 4 minutes. [05/18/2026 18:36] namibj And I count 2.4M lines of node declarations in the earlier resistance-not-including cap-and-devices extraction. [05/18/2026 18:36] rtimothyedwards_19428 @namibj : Make sure you're using magic version 8.3.644, which I updated a few days ago and which specifically resolves an issue with very slow run-times on the R-C extraction. [05/18/2026 18:38] rtimothyedwards_19428 As for the "bad device location" error, I would need a reproducible result. [05/18/2026 18:42] namibj wait, are you _the_ Magic-vlsi guy? [05/18/2026 18:44] namibj I can get the folder bundled up for you into a `.tar.zst` if you want, including the script that did that and the git commits involved in it happening. [05/18/2026 18:45] namibj How "resolved" are we speaking here? [05/18/2026 18:45] namibj https://discord.com/channels/1361349522684510449/1505985508650516561 [05/18/2026 18:50] namibj Yeah it's running with 8.3.581 ๐Ÿ™ [05/18/2026 18:52] rtimothyedwards_19428 Given the rate of development, 581 isn't all that long ago (December), but I did a lot of work on the R-C extraction since then, so it's well worth updating (and yes, I am _the_ magic VLSI guy). [05/18/2026 18:52] namibj {Attachments} 2026-05_media/image-1D4F5.png [05/18/2026 18:52] namibj (the name rung a bell, figured I'd ask to give you the honors deserving) [05/18/2026 18:53] namibj I'll built and use this one for a run then. [05/18/2026 18:53] namibj (that curently running lone one uses the nix setup from @Tholin ) [05/18/2026 18:54] rtimothyedwards_19428 As far as "resolved"---There were two badly-designed nested loops in the R-C extraction process that were potentially causing extraction to be ridiculously slow. I removed one of the nested loops from each process and sped up the extraction time from several hours to about 30 seconds. But that was a specific pathological case and I have no idea if your design is affected by the same problem. [05/18/2026 18:55] namibj this run is one GIGANTIC cell [05/18/2026 18:55] rtimothyedwards_19428 Yes, please send me a tarball and I can look at it. [05/18/2026 18:57] namibj (having things be hierarchical caused the earlier errors and also initial cap extraction before getting to resistance took like 10 or 20 or so hours instead of the like 5 in the current run) [05/18/2026 18:57] namibj ``` drc off gds read {/tmp/tmpk8ys6nzm/chip_top.klayout_editing_03.gds} load {chip_top} select top cell port makeall ext2spice lvs extract do resistance extract all ext2sim labels on ext2sim extresist simplify off extresist all ext2spice extresist on ext2spice resistor tee on ext2spice cthresh 1 ext2spice subcircuit top on ext2spice hierarchy off ext2spice quit -noprompt ``` [05/18/2026 18:57] namibj for reference, the script it's using [05/18/2026 18:58] namibj (generated by a python wrapper that makes a temporary folder for magic to litter it's files into) [05/18/2026 18:59] rtimothyedwards_19428 It is pretty easy to overload the R-C extraction. I generally don't run it on digital standard cell designs. But then I never tried investigating where the process bottlenecks were until a few days ago. I'm sure it can be sped up a lot more. [05/18/2026 18:59] namibj https://wormhole.app/AyyZn5#5lr5k1NHRoty54ncsNhOWQ {Embed} https://wormhole.app/AyyZn5 Wormhole - Simple, private file sharing Wormhole lets you share files with end-to-end encryption and a link that automatically expires. 2026-05_media/social-share-file-37818.jpg [05/18/2026 18:59] rtimothyedwards_19428 The script looks okay but it is not flattening the cell. Hierarchical R-C extraction is "coming soon"---I'm working on it. [05/18/2026 19:00] namibj Yeah I've got me KDAB's hotspot around was about to throw it at it to see what it's doing so long to take over a second per node to split it into two (`.t0` `.t1` and kill the original with the # suffix) and compute the resistance between them. [05/18/2026 19:00] namibj No I flattened in klayout; see the gds here. [05/18/2026 19:01] namibj (It's 675 MiB compressed; 3.56 GiB uncompressed) [05/18/2026 19:04] namibj {Attachments} 2026-05_media/image-961F3.png [05/18/2026 19:05] namibj If you happen to have some suggestion on what to do to it to make it less slow please tell; this is sadly not the most well-described area and as you can tell it takes over 5 hours to get to the point where we'd observe that RC extraction speed. ๐Ÿ™ [05/18/2026 19:07] rtimothyedwards_19428 Maybe don't do full R-C extraction on an entire chip? [05/18/2026 19:08] namibj I might be able to give in a little on the C part but the R isn't really optional here; I'm trying to verify where suspected severe static power dissipation is happening. [05/18/2026 19:09] namibj I hoped one of the commands might be somewhat redundant for this. [05/18/2026 19:11] rtimothyedwards_19428 We don't really have that capability in the open source tools. I think hierarchical R-C extraction is what you're looking for, but it's not there in magic yet. [05/18/2026 19:11] namibj I see. [05/18/2026 19:13] rtimothyedwards_19428 If it's a digital design, then it's really something that the synthesis tools should be doing. [05/18/2026 19:16] rtimothyedwards_19428 I think flattening it and saving it as GDS was probably a bad idea. It should be read in hierarchically so it doesn't take all day to read, then flattened before running extraction. [05/18/2026 19:17] namibj Well, it's mixed; I know pure digital ones don't have these problems; I'm treating this design fault here as an excercise in doing chip-scale Xyce simulation of DC operating points and very brief transient moments; hoping to get to a point where one can inject a gate level digital simulator's register state as initial conditions/`.nodeset` into the DC operating point run to at least be able to simulate select events from the chip state they would happen in. [05/18/2026 19:18] namibj Ohhhhhh, I see. Yeah, I'll try next time. Actually LVS extraction takes abotu as long as the initial read anyways; they were around 90 minutes each. [05/18/2026 19:19] namibj oh sorry actually loading only took 20 minutes [05/18/2026 20:07] rtimothyedwards_19428 For me, the loading has been going on for > 40 minutes and looks like it is being slowed down by one specific "grow" operation. So the size of this layout has exposed some other inefficiency that I'll also need to look into. [05/18/2026 20:17] namibj ahhhh [05/18/2026 20:19] namibj Hope you don't see this as a bad thing but rather as a healthy benchmark in scalability. Any idea how feasible parallel extraction might be for situations like this? I'd guess that given the already tile-based method (there are notes about the tile size needing to be tuned for the tech to optimize performance as both too small and too large would be slower than optimal size), it shouldn't be too hard in theory, aside from the Tcl engine being from what I can tell quite single-threaded in it's nature. [05/18/2026 20:27] rtimothyedwards_19428 I was working on parallelization recently. There are several places where the database gets altered during a search, which prevents two processes from accessing the same data. I got rid of one of them, haven't worked on the other one yet. After that, yes, it should be possible to parallelize a lot of things in magic. [05/18/2026 20:27] namibj I'm asking because Xyce will scale to MPI clusters at least a rack in size quite happily it seems for designs of this kind of size. [05/18/2026 20:28] rtimothyedwards_19428 Well, Xyce was designed specifically for parallel processing. Magic was not, and I have to do the footwork to modernize it. [05/18/2026 20:28] namibj What programming language is that work done in? I might be motivated to help; profile-guided optimization is one of the larger things I've been doing fairly well at in past years. [05/18/2026 20:29] namibj I meant that as there being open tooling to consume the large PEX netlists that arise from full-die extraction. [05/18/2026 20:32] rtimothyedwards_19428 Magic was written in 1985, so it's all in plain C. [05/18/2026 20:33] namibj (I figured there might be substantial Tcl involved in those parts, but plain C is good for this.) [05/18/2026 20:34] rtimothyedwards_19428 There is a lot of Tcl, but not in the core code. [05/18/2026 20:35] namibj Yeah in that case I might be able to help somewhat. Let's see how this run progresses at least. [05/18/2026 20:36] rtimothyedwards_19428 Any help would be appreciated. Especially profiling. [05/18/2026 20:40] namibj yeah, I'll try to get the next build with debug symbols so DWARF unwinding will work on it [05/18/2026 20:40] namibj well, better debug symbols [05/18/2026 22:10] namibj oh yeah it feels like you _might_ have accidentally broke loading speed. I _thought_ it was further along with the december version, but we'll see if it went through to extraction after I have slept. [05/19/2026 20:48] rtimothyedwards_19428 In the end, the loading ended up being relatively well balanced among the input planes magic was generating. They were pretty much _all_ slow to read. But that means it would benefit greatly by parallelization, since there didn't appear to be any single plane taking up all the compute time. I ran overnight and magic generated a 2.5GB .ext file. Since then, it has run all day and the .res.ext file is still zero bytes. That's not entirely surprising; when it hits either the common vccd or vssd networks, it's going to slow to a crawl, because those are going to be massive networks. [05/19/2026 20:52] rtimothyedwards_19428 But I'm analyzing it as it runs and finding all the chokepoints. [05/19/2026 20:59] namibj same [05/19/2026 20:59] namibj but in the older version it started filling the resistance file sooner [05/19/2026 20:59] namibj so you're getting profiling information from this at a macro scale at least? [05/19/2026 21:01] rtimothyedwards_19428 It's not actually profiling; I'm just spot-checking what magic is doing in gdb, and it's pretty obvious when any arbitrary Ctrl-C lands in the same routine that there's a bottleneck there. [05/19/2026 21:02] namibj `/nix/store/qwianildz86dh9yhmjbj250a7qjlp7pk-magic-vlsi-8.3.581/lib/magic/tcl/magicdnull` has started writing to `.res.ext` within about 12 hours oro so. [05/19/2026 21:02] namibj ahh kk [05/19/2026 21:03] namibj if you have debug info to that extend I suggest you try KDAB's hotspot [05/19/2026 21:03] rtimothyedwards_19428 "but in the older version it started filling the resistance file sooner"---Output is buffered, so if it hits either the power or the ground net before the buffer is full, output will remain zero for a _long_ time. If it processes enough smaller nets first, then it reaches the buffer limit and the output file grows. It's basically random behavior. [05/19/2026 21:03] namibj I'd get youa trace from thiscurrent run if I had enough debug symbols to get any line information and not have half the call stack be ???? symbol [05/19/2026 21:04] rtimothyedwards_19428 I am only spot checking because I wasn't really intending to do a proper profiling on this run. I just wanted to see what it was doing at a very high level. [05/19/2026 21:04] namibj oh yeah it spits out in like 8 kB chunks [05/19/2026 21:04] namibj hotspot attaches to a running process just fine if you can attach gdb [05/19/2026 21:05] rtimothyedwards_19428 Anyway, I already found a bottleneck and have a solution for it. There's a routine that's trying to sort resistors by size by walking a linked list. It would benefit immensely from qsort(). [05/19/2026 21:06] namibj it uses kernel based sampling to capture stack top snapshots for post-processing unwinding via DWARF (or if youhave frame pointers, live unwinding via those), and gives you nice timeliene and flamegraph understanding; it's really nice. [05/19/2026 21:06] rtimothyedwards_19428 I will give it a try. [05/19/2026 21:06] namibj oh yeah probaly. [05/20/2026 01:43] rtimothyedwards_19428 Magic spent the entire day in the sort routine, so I killed the process, rewrote the routine, and confirmed that it shaves a few seconds off extraction in another (much smaller) test case. I will re-run extraction on your chip overnight and see where it is tomorrow. [05/20/2026 03:01] namibj if it seems to make decent progress share please; for reference this is what I've used to build; https://aur.archlinux.org/packages/magic (see for the build "script") [05/20/2026 03:04] namibj if you have whatever particular config you're using to have the debug symbols properly working do tell; I hope I don't need particular debug symbols for tcl to see this in the stack trace though, but I'd get those myself if I need to. [05/20/2026 03:06] namibj also if you need to credit an author of the file that'd be more @Tholin than me; I only flattened it and erased port information from anything but the pads and disambiguated the pads so I can later drive them with packaging/board parasitics at least _somewhat_ modeled. [05/20/2026 13:37] rtimothyedwards_19428 It ran overnight and has already moved much further than it ever did yesterday; it has so far produced a 350MB `chip_top.res.ext` file. So far the only content in the `.res.ext` file is the VSS net, which has around 2 million nodes in it. So it is slow but appears to be methodically making its way through the extraction. [05/20/2026 16:40] namibj Share code pls. [05/20/2026 16:58] namibj @Tim Edwards `ResTriangleCheck` Is searching for triangles in an undirected graph, right? That is O(n^(3/2)) not O(n^2); I'll see about fixing that aspect in a few weeks if you remind me. [05/20/2026 17:59] rtimothyedwards_19428 It can be O(anything) if it's written badly enough. {Reactions} ๐Ÿคฃ [05/20/2026 18:04] rtimothyedwards_19428 The first set of fixes can be found in the magic source code on github now. I made another fix today to speed up a pass through the contacts, and I will push that code later today. [05/20/2026 18:21] namibj No I mean triangle counting/enumeration is O(n^(3/2)) worst-case if written sufficiently non-bad assuming hash tables; and like a factor of `(log2(n))^2` worse at most if not using hash table base. [05/20/2026 18:23] namibj n being edge count [05/20/2026 20:58] namibj @Andrew Wingate see, I've done substantial research into the direction of applied (beyond) worst-case-optimal join algorithms. The mentioned triangle search can be made closer to `O(n)` than naive `O(n^2)` (to a worst case of `O(n^1.5)` aka `O(n^(3/2))`) by: 1. As resistors are undirected edges, roughly presume we just want each resistor in both directions in any index structures (minor improvements can later be made). 2. Create an index giving us a list of target nodes for each source node (the resistor metadata is carried as auxiliary data with each list entry). 3. If needed for some access efficiency use a separate index (otherwise use the same and keep a tuple of `(list((target_node, resistor_identity_metadata)), length(that_very_list))`) to quickly look up how much fan-out happens from grabbing that list from a given key into the first index. 4. Iterate through the list of edges. 5. For each edge, look up both the first and the second node of that edge in the length-index (the second index, from 3.) and decide which one has less fan-out. 6. Look up the one with less fan-out in the first index (from 2.) to get the worst-case-optimal choice of what the second edge is and what the third edge will be for specifically the current iteration (first edge) of (4.). 7. Screen each such-fetched triangle node-triple, specifically the two nodes still dangling (i.e., the non-chosen one from the first edge and the target node of each entry of the second edge), against a hash table or similar set membership test data structure to spend only constant time on each candidate of the third edge (this I think is currently being done since the upgrade to `O(n^2)`; the major trick is just to dynamically choose the less-costly vertex aka join-order for each first edge). 8. You have successfully completed the task of enumerating all triangles in at most `O(n^1.5)` time. I'll see to get profiling up so I can hopefully run PEX against a similarly edited GDS of the TinyQV quarter cell submission that "only" has 4 VSS and 4 VDD bond pads and is, well, a quarter the die area (usable area not just filled with power rings/IO cells is smaller), in hopes of that GDS being large enough to exercise poor scaling but small enough to not take as many hours to even just load (such that we could reasonably use it for full profiling runs in the coming performance/throughout enhancement updates I hope to contribute greatly to). {Reactions} ๐Ÿ’œ waferspace [05/20/2026 22:37] namibj So, we're in luck, I managed to get frame pointer and debug symbols compiled in for a build of `8.3.645` and am running on `tinyqv_quarter_label_flattened_01.gds` (and I choose a cutoff of 0 for the capacitance as that happened only later and I figured I'd rather get this done now and let Xyce worry to possibly threshold that rather than erase the smaller capacitances ahead of it when magic writes out the extraction in the desired spice netlist format) currently with a `cycles` perf record running with stack tracing using said frame-pointer's; I confirmed it works nicely in KDAB's hotspot for me. Might have to adjust source file search path I guess, will see in a bit. Figured I'd upload the gds and also the edited but not yet flattened gds here, as they're small enough when compressed. {Attachments} https://media.githubusercontent.com/media/wafer-space/discord.wafer.space/master/large_media/wafer-space/Information/questions/is-magic-pex-extraction-expected-broken-for-gf180mcufdscmcu7t5v0fillcap64/2026-05/2026-05_media/tinyqv_quarter_label_flattened_01.gds-8B335.zst 2026-05_media/tinyqv_quarter_label_edited_01.gds-BD15C.zst [05/20/2026 22:38] namibj At the current growth rate of the perf recording I've got about 7 days until the SSD is filled up; but as this is compressed and contains stack traces it will be vulnerable to changes in data rate as the process stages changes. {Reactions} ๐Ÿ’ฏ [05/20/2026 22:51] namibj for your pleasure/sneak peek; a short second recording done while it was running: {Attachments} 2026-05_media/image-A1BC4.png [05/20/2026 22:52] namibj flame graph filtered down to `extEnumTilePerim (tclmagic.so) and below.` [05/20/2026 22:54] namibj the blue timeline sections are the non-hidden flamegraph content; i.e., in the orange timeline sections it was doing _not `extEnumTilePerim`_ [05/20/2026 22:55] namibj notably it was doing `extBasicOverlap` instead of `extBasicCouple` (the latter the direct parent of `extEnumTilePerim`). [05/20/2026 22:57] namibj @Tim Edwards if you want I'll give you the profiling result from this run when the run concludes; though for reasons of possible contamination with trace amounts of personal data that one won't be public data (I don't think it has gobbled up any of that, but it's hard to rule out). [05/20/2026 23:13] rtimothyedwards_19428 @namibj : I think it is better to wait until the changes I made today to magic migrate to github (in about 8 hours). I found more routines which it was spending long periods of time on. I changed the code in both places to cut a significant amount of time off of the execution. I'm re-running everything for the (I think) fourth time. The last routines I fixed were pretty far into the "extresist" process, so there's a good chance that there are no more obvious outliers taking up all the compute time. I think that the initial GDS read-in time could be significantly reduced by being careful about what order layers are processed in the "cifinput" recipes in the tech file. If the recipe has "A AND B AND C" where A and B are very common layers and C is a very rarely used layer, then "A AND C AND B" will execute much faster. [05/20/2026 23:14] namibj Hmmmm [05/20/2026 23:16] namibj This one is a quarter the size and substantially less dense as the usable area is substantially less than a quarter; it doesn't seem to really have analog but it's from the same MPW run so should be sufficiently represenative, I hope, for purposes of trying/iterating. Do consider using that until you think it's free of any easy to fix and severe performance bugs. [05/20/2026 23:19] rtimothyedwards_19428 Okay, thanks. I'll wait for this run to finish, or at least see where it's at tomorrow morning if it hasn't. For future runs I'll switch to the quarter size layout. [05/20/2026 23:21] namibj Yeah ok. (What's blocking the changes from migrating faster to a place I could grab the sources from and build to run the profiling on?) [05/20/2026 23:21] rtimothyedwards_19428 Oh, if you want the sources right away, just get them from my server at opencircuitdesign.com. I use github as a mirror and do an automatic copy every night by cron job. [05/20/2026 23:22] namibj Ohhh ok. [05/20/2026 23:22] rtimothyedwards_19428 (`git clone git://opencircuitdesign.com/magic`) [05/20/2026 23:26] namibj > STATUS: Partly done (early 2020). Extresist now handles all device types, but there is a very strong need to remove the dependence on reading the ".sim" format and just work all calculations from the values in the ".ext" file. For example, extresist cannot now handle the substrate node because the ".sim" format has no concept of a substrate. oh, would this enable bulk substrate current flow treatment for PEX? At least, in theory; considering that the substrate being substantially thicker than most features would invalidate the simplified sheet resistance 2D-only treatment done so far on BEOL PEX? [05/20/2026 23:34] namibj ok running that then; compiling. [05/20/2026 23:52] rtimothyedwards_19428 Actually I resolved the issue with the substrate before I removed the dependency on the .sim format. The main thing that removing the .sim dependency did was that it allowed me to start working on hierarchical R-C extraction, which I have partly completed work on (but haven't pushed yet; still just a local branch on my desktop). [05/20/2026 23:53] namibj Did the sim dependency get removed some time since december? [05/20/2026 23:54] rtimothyedwards_19428 Yes. Most of that work was done in January, I think. [05/20/2026 23:55] namibj This hierarchical R-C extraction you speak off, how does that handle the layer-penetrating capacitive coupling in, let's say, a standard cell fully-synthethic floorplan (a plain hard macro synthesized from Verilog into a prescribed rectangle; let's presume it goes all the way to top metal)? [05/20/2026 23:55] namibj Like, between the cells themselves and the sea of cells. [05/20/2026 23:56] namibj Yeah I noticed the files that got produced changed since I moved from the old nix-provided magic version in these PEX trials here. Edit: and just was a little curious now that I read about sim file dependency getting removed. [05/20/2026 23:58] rtimothyedwards_19428 The parasitic capacitance is already taken care of, hierarchically. The problem is distributing the parasitics correctly to each sub-network where a net has been broken down into resistive components. The other problem with hierarchy is determining for each connection between parent and child, which subnets that connection is between. [05/21/2026 00:01] rtimothyedwards_19428 The capacitance calculations are quite complicated, looking at the child cell in isolation, the parent cell in isolation, then the parent cell with each child cell added in turn, being careful to find where parasitics in isolation get modified by the presence of all the child cells. [05/21/2026 00:41] namibj Yeah, to me that has seemed overall somewhat wasteful? Like, I don't _really_ get how that will benefit simulation. At least Xyce appears to just flatten at parse time anyways, where this may at most have vestigial effects on how the circuity is distributed across parallel compute nodes if doing some netlist-ordering-based distribution tactic. To me it seems like it should probably be usually cheaper computationally to just extract the PEX flattened, or at least the coupling part (resistance could probably be extracted locally with little harm, but would probably have to treat each contact spot of the ports independent to use the correct node to tie the higher level resistance to the intra-cell resistance). That said, the capacitance extraction should be pretty straight-forward to do in parallel, just tile the overall area into twice as many strips as one wants to run in parallel, keeping a minimum strip width large enough to completely absorb the sideways fringing modeled reach, then process all even strips first in parallel, then add all odd strips in a second parallel run. [05/21/2026 00:47] rtimothyedwards_19428 It doesn't benefit simulation. It benefits extraction, which is quicker, and produces much smaller files. [05/21/2026 00:50] rtimothyedwards_19428 Also: Since magic's `.ext` file output is an intermediate form, and all the hard work is done to get to the intermediate form (`ext2spice` is quick and easy), then the hierarchical format also has the benefit that from the intermediate `.ext` form, you can quickly get any netlist for any cell in the design, with or without parasitics, without having to go through the extraction process again. [05/21/2026 00:51] namibj Hmmmm.... [05/21/2026 00:55] namibj {Attachments} 2026-05_media/flamegraph_1-91661.png [05/21/2026 00:58] namibj seems to be from this region; it was stacktraces that got chopped off for being too deep. {Attachments} 2026-05_media/image-4067E.png [05/21/2026 01:00] rtimothyedwards_19428 Except that all that is from basic extraction; that's been worked over quite a bit and I consider it to be reasonably efficient. For example, on my desktop, the extraction of your full chip design took 1 1/2 hours. Right now, the vast majority of the compute time is not in the basic extraction but in the resistance extraction, which happens afterward. [05/21/2026 01:06] namibj Ahhh yeah it's getting there. [05/21/2026 01:07] namibj (That was more a questioning of how wise of a decisiion it is to do that stepping recursively; it feels like it might hit stack exhaustion on some layouts due to that.) [05/21/2026 01:09] rtimothyedwards_19428 I have gotten rid of most of the methods in magic that were apt to hit recursion limits. I know the extraction "walk" routines get rather deep, but I have not yet encountered anything that hit a stack limit and crashed. [05/21/2026 01:11] rtimothyedwards_19428 Note that pretty much all of these routines were written in the 1980s. The "wisdom" involved is usually about deciding whether to spend the time to analyze the method and come up with a better one, or to work on something higher up on the priority list instead. [05/21/2026 01:14] rtimothyedwards_19428 Right now, the next chokepoint I'm seeking is the loop at `resis/ResMakeRes.c` line 395. [05/21/2026 01:21] namibj Oh yeah wasn't suggesting it's necessarily a problem; but yeah. Didn't know how aware you were as it seemed to not be that much of the time actually spent in the severe recursion cases; it just became visible to me because I did so much sampling; it reports on the order of 1.4e9 cycles to have been spent in that highlighted deep region; I think it was a bit more timeline-restricted than quite fair, so probably maybe 5~10 seconds of CPU time spent in such deep recursion that it blew through the unwinding's depth limit. [05/21/2026 04:12] namibj That linked list chase has shown to take non-insiginificant time froma brief profile capture. Notably, most time is supposedly spent in the pointer fetch of the if's `cmp` assembly instruction. [05/21/2026 04:16] namibj I'm assuming most of that linked list choice exists for mostly historical reasons due to being fairly old and the cost of these cache misses not being nearly as severe (relatively speaking) back then. Same btw. for EastWest. [05/21/2026 05:17] namibj I'll have a good hour or two of profiling right in the middle of vss net resistance extraction (and probably going on to vdd or whatever nets it chooses next) for the last of the 4 GND bond pads to look at after a brief nap, though not for too long sadly, before I have to go to the office and do $dayjob things. At this pace I do have hope that the extraction will finish within the next 24 hours and I can then get the files dragged together for you to enjoy the profiling recording of the entire run on the `0.5x0.5 TinyQV flattened` gds. I think there was some tool to bundle all the files needed to make unwinding work for you, but I'm sure I'll find a decent way to make it viewable in KDAB's hotspot for you (including decompilation view!) until then. I'll plan to send you a .tar.zst and necessary instructions for how to feed the data to hotspot via wormhole as discord DM; let me know if that's a problem. [05/21/2026 07:49] namibj Also I don't know if you've heard of it before, but as far as attaining a _quantified_ and attributed understanding of `malloc`-spam goes, [KDE's heaptrack](https://invent.kde.org/sdk/heaptrack/) (from the same author as [KDAB's hotspot](https://www.kdab.com/software-technologies/developer-tools/hotspot/)) has done wonders for me when I used it on some photogrammetry/structure-from-motion toolchain [TU Darmstadt's MVE](). Small commits like (heaptrack), (hotspot), or (iirc this one was highlighted by heaptrack for suspicious allocation intensity, then hotspot was used to get a quantified understanding of how bad ||it seemed to be to try and estimate how much whole-program speedup could be reasonably expected to be attained, before I looked for a suitable (smol, easy, compatible licence) concurrent union-find library to "easily" vendor into the project and swap out the BFS implementation of the connected-components task for a union-find one. See below for excerpt of the changed file that should make obvious how this kind of problem could be surfaced by heaptrack.||), are representative of the impact I managed with those 7 years ago. ```cpp /** * A complet contains a partial component: a list of node, its temporary * component ID and the ID of seen other complets */ using complet = std::tuple, luint_t, std::set>; ``` ```cpp tbb::concurrent_vector accrued_complets; ``` {Embed} https://invent.kde.org/sdk/heaptrack/ SDK / Heaptrack ยท GitLab A heap memory profiler for Linux 2026-05_media/avatar-D0BA8 [05/21/2026 09:14] namibj {Attachments} 2026-05_media/flamegraph_4-E4F8D.txt 2026-05_media/flamegraph_4-98EF8.svg 2026-05_media/flamegraph_4-9EBA7.png [05/21/2026 09:16] namibj This is from two big nets (must be one or two vss and one or zero vdd); I filtered carefully but limited by pixel-perfection to the time range of these two among the over 4 hour long `perf record`'s recording. It's CPU cycles spent while executing the process in user mode. [05/21/2026 09:17] namibj the txt is what it calls "stack collapsed" export. [05/21/2026 09:17] namibj The 3 files are otherwise all from the very same filtered view. [05/21/2026 09:19] namibj also, just fyi {Attachments} 2026-05_media/log_so_far_2026-05-21T09_19_0300_00-F029B.txt [05/21/2026 09:20] namibj that's after 9h35 min of CPU time on the process. [05/21/2026 09:20] namibj (I just re-used the local PDK copy from the other (the large) chip.) [05/21/2026 15:56] namibj bad news: it crashed at the end. good news: it appears to have finished at least almost all steps. {Attachments} 2026-05_media/log_so_far_2026-05-21T15_53_4600_00-091BE.txt [05/21/2026 16:03] namibj {Attachments} 2026-05_media/image-CF218.png [05/21/2026 16:03] namibj {Attachments} 2026-05_media/image-7017C.png [05/21/2026 16:03] namibj {Attachments} 2026-05_media/image-3422C.png [05/21/2026 16:04] namibj {Attachments} 2026-05_media/image-CABA7.png [05/21/2026 16:07] namibj That _looks_ like it may be the `Bad Device Location at 210074,478892`; it lines up nicely with what seems like it could be a device's bottom left corner, and has that 5/1 scaling applied that magic spoke up at the beginning of the log. [05/21/2026 16:17] namibj `hotspot 1.6.0`; the `.perfparser` pre-digested recording comes soon. It's processing. [05/21/2026 16:20] namibj (It looks to be taking about 20 minutes to process through `/usr/libexec/hotspot-perfparser`) [05/21/2026 21:32] rtimothyedwards_19428 Meanwhile I worked around the issue at ResMakeRes.c:397 with a hash table so that it can convert nodes on the main pass through the linked list instead of running a second loop through the rest of the list to search for nodes that need converting. This time I will run on your quarter-size chip so that I can see results by the end of the day, maybe, and also take a look at that crash condition. [05/21/2026 21:42] rtimothyedwards_19428 Sorry for talking past you somewhat here. . . It's just that there are obvious chokepoints that need to be fixed, and I don't really need a flame graph to figure out what they are and what to do about them. I think I'm quickly getting to a point where the resistance extraction is decently efficient, at least for now. [05/21/2026 22:24] rtimothyedwards_19428 Also it looks like the next performance bottleneck is the use of DBPaint() and DBErase() in ResDissolveContacts(). It should not be using such high-level paint routines; it should be pretty easy to cut a large amount of time off of those calls. [05/21/2026 22:35] namibj Oh sorry yeah I mostly collected it as I tried to get output from it (on the quarter slot) at the same time; it failed extracting a device as noted which seemed.... strange. But hotspot allows masking out parts both in timeline view and in flamegraph view (arbitrary combinations) via "filter out symbol" to jump to the point where these smaller improvements are already presumed-done (at least the fundamental one like the hash table), so one doesn't need to wait for a run to complete just to get an understanding of what's the next most relevant part. [05/21/2026 22:37] namibj You might want to attempt to understant what's going on with the device extraction failure if you want to see performance of the SPICE generation at the end. I just put all 3 versions of the quarter slot gds I have here in case I missed any. {Attachments} https://media.githubusercontent.com/media/wafer-space/discord.wafer.space/master/large_media/wafer-space/Information/questions/is-magic-pex-extraction-expected-broken-for-gf180mcufdscmcu7t5v0fillcap64/2026-05/2026-05_media/tinyqv_quarter_label_flattened_01.gds-E9F16.zst 2026-05_media/tinyqv_quarter_label_edited_01.gds-93BC4.zst 2026-05_media/tinyqv_quarter.gds-37FE9.gz [05/21/2026 23:17] rtimothyedwards_19428 Nothing like a full chip design to exercise all the corner cases. . . [05/21/2026 23:46] rtimothyedwards_19428 @namibj : The full R-C extraction of `tinyqv_quarter_label_edited_01` completed on my desktop in just under two hours. I did not get a crash condition. I did, however, encounter two instances of `Missing rptr at (...)`, indicating an attempt to access a resistor structure that had been previously freed, and a bunch of `Missing substrate conection of device at (...)` which remains to be debugged. I did not get any `Bad Device Location` errors. [05/21/2026 23:47] rtimothyedwards_19428 The final netlist is just under 2GB. [05/21/2026 23:47] namibj `PDK_ROOT=`realpath gf180mcu_pdk` PDK=gf180mcuD scripts/extract_spice.py /home/namibj/Downloads/tinyqv_quarter_label_flattened_01.gds --cthresh 0 --subcircuit-top --hierarchy-off` {Attachments} 2026-05_media/extract_spice-C9D1F.py [05/21/2026 23:48] namibj something Xyce or ngspice would digest? [05/21/2026 23:50] rtimothyedwards_19428 BTW, `Missing substrate connection` means that magic could not determine what subnet a device's substrate (or bulk terminal) connects to, so while it does not exactly invalidate the netlist, it means that some devices will have a substrate connection to the default, which would be the label at the pad for one of the VDD or VSS pins, and not somewhere deep inside the substrate's resistor network where it should be. The final output SPICE _might_ be digestible by Xyce. Not sure how ngspice will handle a 2GB netlist. [05/21/2026 23:50] namibj if there's anything convenient in particular I should add to get step progress information in the future I'd be happy to add that [05/21/2026 23:51] namibj ngspice was more of a "in theory it meets the spec" part; I'm squarely looking at Xyce for this type of job. [05/21/2026 23:53] rtimothyedwards_19428 This is the shell script I'm using to do the extraction. It is hard-coded to my PDK location and the GDS file and top cell name. {Attachments} 2026-05_media/run_tinyqv_rcx_extract-C26FE.sh [05/21/2026 23:53] namibj The above linked script's output on designs like a TT submission or in this case, iirc Tholin's DAC hard macro, resulted in a valid netlist. [05/21/2026 23:54] namibj I'm positive the prior version which to my eye had identical tcl fed to magic aside from the obvious path/name differences produced functional ngspice and Xyce netlist. [05/21/2026 23:55] namibj thanks for your script [05/21/2026 23:56] namibj (I'm about to sleep though, so best of luck to you and big thanks already.) [05/21/2026 23:57] namibj As for the substrate connection, I think the sealring was isolated on all layers both to the inside and the outside of the chip. I think some ESD diodes in IO pads may also expect implicit substrate connectivity. [05/22/2026 00:00] rtimothyedwards_19428 I am on travel from some time mid-day tomorrow, so I might not get back to work on it for a few days. [05/22/2026 00:00] namibj If it's just a 2-hour run I'd at least strongly consider throwing mozilla's rr at it, fast forward in the replay to the missing rptr notice via probably a breakpoint set to where the notice is emitted, then use reverse-continue with data watchpoints on the allocations that got freed before. Your deep knowledge ofc may bring a more specific path of debugging that to light. [05/22/2026 00:05] namibj That's no problem gives me time to port my MCML VCO efforts from sky130A to gf180mcuD to see where the speeds and sizing are approximately at; such that I can get the PCell topology adjusted and at least hope to make it for the RUn2 deadline even though I failed to make it for the ttsky26a deadline (barely got two instances of a DNL-compensatable-via-digital-calibration DAC onto it, instead of the supposed DAC-tuned PLL/VCO core). [05/22/2026 16:07] rtimothyedwards_19428 I pushed another update in which I got rid of the calls to DBPaint() and DBErase() in favor of simpler lower-level subroutines that do the same thing more efficiently. The total time to process the quarter chip is now down to just under an hour and a half. The process is now spending much of the time doing connectivity searches, which is already highly optimized, so I don't think there will be too much further opportunity to cut significant percentages off of the processing time. [05/22/2026 17:29] namibj ok [05/25/2026 02:04] rtimothyedwards_19428 I found another issue in the code and knocked down the time from 1 1/2 hours to 1 hour (so I cut another 50% off of the extraction time). (Version 8.3.649). [05/25/2026 18:53] namibj Yay. (I'm sorry the heatwave has impacted my productivity.) [05/25/2026 20:54] rtimothyedwards_19428 You're way overproductive and the heat wave is allowing me to catch up. : ) [05/25/2026 23:03] mithro_ Random fact, US circuits suck, so the only place I could plug in my server that provided enough power was the power point for my AC (which was 240V). I either had A/C or a 2400 watt heater (IE the server), great for winter, terrible for summer ๐Ÿ™‚ I ended up building a mini-datacenter to deal with the heat and stuff.... {Attachments} 2026-05_media/AP1GczP81uPF2LpfXV3qBbqxBWsXeP9BRI1UAb5R-Y-DFCB9.png 2026-05_media/AP1GczMS15Yv7UuQCuK3lNgI5-bdpW7mz-TZ3W1vdv-B2B79.png 2026-05_media/AP1GczM9eOdA4ZhZYqiaXbi_1k5IVWUPxkvGlmkC5M-0BF46.png [05/25/2026 23:17] rtimothyedwards_19428 I visited Tim in his apartment and I can attest to the fact that he was basically living in a server closet. [05/25/2026 23:20] rtimothyedwards_19428 I can also attest to the fact that many (most?) houses and apartments in the Bay Area are built without either insulation or air conditioning, and when it hits 38C in San Jose (which is fortunately rare, but it does happen), it is a bit like living in an oven with no door. [05/26/2026 03:49] namibj I'm turning my shed into my data center, though it'll be fresh air intake (filtered) with just hot/cold aisle separation. [05/26/2026 03:52] namibj Gotta knock out the SAS and power cable hole between the two chassis and 3d print a baffle up front to make the top one suck it's air through the hdds caged in the bottom before hitting the ssds and then the fans and then the ram/CPUs, before meeting the rest. [05/26/2026 08:53] mithro_ My dad is helping me turn an old shed into that right now too! {Attachments} 2026-05_media/get_attachment_url-A151F.png {Reactions} ๐Ÿ’œ [05/26/2026 21:12] rtimothyedwards_19428 @namibj : I looked into the problem with the "missing substrate" terminal detection, determined that it happens specifically on diodes, and specifically because diodes have 2 or 3 terminals, and there is code for "extresist" that still expects all devices to be 4-terminal FETs. I previously (a long time ago) added some lines that ignore devices that have fewer than 4 terminals, as a way to avoid the problem while working on other things, but I forgot that was in there, and the handling of diodes has never been checked. I started fixing that but am running into segfaults, and it looks like a not so easy problem to track down the places where two or three terminal devices are not being handled correctly. [05/26/2026 21:22] namibj oh. I'm unfortunately too busy to hunt down those memory bugs until towards end of next week, sadly. In similar extraction spirit, do you happent o have _any_ knwoledge of 3v3 native nmos existing on this process, and if so, whether there might be something vaguely-practical to mash existing nmos_03v3 and nmos_06v0_nvt models together considering that they at least claim to be physically-based and the only physical substantial difference should be the oxide thickness that's deposited? At least that's the only explanation I could come up with that wouldn't involve an extra physical mask to handle 3v3_nvt in addition to 6v0_nvt. [05/26/2026 21:41] rtimothyedwards_19428 @namibj : (1) No worries; I am working through the problems and was just letting you know the status. (2) Yes, assuming that the NAT layer is included in the process options (which I think is true for Wafer.Space), then both 3.3V and 5/6V native nMOS devices exist. That's according to the documentation: "This process supports 3.3V and 5V/6V native Vt NMOS. The difference between 3.3V and 5V/6V native Vt transistor is that the 5V/6V native Vt NMOS need Dualgate layer." The real problem here is that there is no _device model_ for the 3.3V native Vt nFET. Given some test structures and somebody with a wafer probe station and the proper instrumentation, we could presumably generate a reasonable device model. [05/26/2026 22:17] namibj As we have models for the other siblings and those models are claimed to be physical models, would it be reasonable to interpolate into the `nfet_03v3_nvt` and hit approximately as (in-)accurately as the regular process corners (ff/fs/sf/ss) already are? I'd try to make it for this tt experimental shuttle at least to get the MCML VCO + NRZ serialiser completed drawn, and I'd quite like to use them in the upper level of the D-latch DIV2; bias current is externally supplied anyways and should cover a lot of potential variation, it's only really the strength of the memory pair vs. the input pair that really matters due to the tail current source so yeah... Honestly I am severely surprised they just left it out? Completely? I'd think we should at least get some etest data, that has to exist, right? [05/26/2026 23:36] namibj I haven't tested them _yet_ but that'll not be too long. Even if they don't model/simulate, they will have to extract even just for LVS to be possible; I believe the current state of silent swallowing /ignoring is the worst option; extracting a model that clashes for lack of spice library coverage should still be better than silently ignoring what to our understanding will become a non-negligible transistor. Oh, and just for completeness's sake, the gds I drew real quick in klayout with the `nfet_03v3_nvt` device structure that doesn't extract. {Attachments} 2026-05_media/nfet_03_v_3_nvt_bins_6_10-42483.spice 2026-05_media/nmos_03v3_nvt_test_flattened_01-95E66.gds [05/26/2026 23:38] namibj (Oh, yeah, disclaimer almost forgotten; that spice there was LLM'd out of the `nfet_06v0_nvt` and the `nfet_03v3`models.) [05/26/2026 23:40] rtimothyedwards_19428 I can add a device model extraction to the tech file for magic. [05/26/2026 23:40] namibj Thanks. NO rush I won't get to use it before middle next week at the earliest. [05/26/2026 23:42] namibj If you dare look at the LLM-written .spice, I wouldn't hate an opinion on the uncertainty treatment/coding done in there; that was not the LLM's idea to be clear, that was my thought. [05/26/2026 23:43] namibj well, not get to use it for anything productive, at least. I might be able to run it sooner ๐Ÿ˜„ [05/26/2026 23:43] rtimothyedwards_19428 @Tim 'mithro' Ansell : Any chance of getting hold of a model for what should be `nfet_03v3_nvt`? It did not exist in the original GF PDK. It probably exists in other GF180 processes, where the model might even be similar enough to use as a placeholder until we can characterize the device ourselves. Getting GF to open source that model is a separate question, though. [05/26/2026 23:47] namibj honestly some etest data ideally on a setup that also has `nfet_03v3` and `nfet_06v0` and `nfet_06v0_nvt` under presumed-same process corner and with sufficiently similar probing parasitics should allow narrowing in these uncertainties of the fairly-naive extrapolation into this missing model "corner"; it'd probably be functional after that. We already have all the sibling models in the PDK, after all; I don't see what beyond etest qualification ground-truth is missing to properly fit that model corner.... [05/26/2026 23:50] rtimothyedwards_19428 @namibj : The SPICE model for `nfet_03v3_nvt` should not have parameters `s_sab` and `d_sab`. Those parameters exist only for the "drain-side (un)salicided" devices which have the suffix `_dss`. The `dss` devices are very different and the `nvt` device should not be derived from them. [05/26/2026 23:50] namibj (If I'm wrong please do tell. And I don't particularly see why they would have good reason to hide a dump of etest for the 4 symmetric nfet devices on gf180mcuD; unless they don't casually collect such data or the way they collect it involves too-secret transistor geometry choices.) [05/26/2026 23:51] namibj Oh thank you I will get that taken care of. [05/26/2026 23:51] namibj (But the general concept of the uncertainty modeling is not-bad, or?) [05/26/2026 23:54] rtimothyedwards_19428 No idea, really. I have to stare at the models for a lot longer. [05/26/2026 23:54] namibj oh. [05/27/2026 00:00] namibj Basically the idea was to instead of the normal mostly-2-dimensional "process corner" scheme (very separate from local missmatch) to add a bunch more dimensions to cover the independent knobs of the BSIM model that had to be mostly-guessed/extrapolated from the 3 other `nfet` variants, with the goal being that if one can at least give decent estimations for the range of values that would be plausible/realistic to have there given the other nfet models, then the monte carlo tactic would allow testing if across the ranges of realistic parameter values, the circuit in question would nonetheless stay functional (even if perhaps somewhat derated from the plans), which would allow using it quite liberally in the ttgf0p3 analog test die that goes in Run2. [05/27/2026 00:02] mithro_ Any idea why it didn't exist in the original GF PDK? [05/27/2026 00:02] rtimothyedwards_19428 No clue at all. [05/27/2026 00:05] namibj even `nfet_06v0_nvt` is only a singular bin..... also doesn't appear to have any missmatch modeling done to it. [05/27/2026 00:06] namibj (I'm low key surprised you're still paying for the `NAT` mask layer on the upcoming runs, but I guess it's easier than modifying the PDK tooling to get rid of it and change what exdact process is run at glofo.) [05/27/2026 00:07] mithro_ From https://bit.ly/gf180mcu-names - I'm guessing it would be called something like nmos_3p3_nat in the original PDK. It does seem like it appears in the ULL and IC PDKs {Embed} https://bit.ly/gf180mcu-names GF180MCU (GlobalFoundries 180nm PDK) -- Primitive Model Naming [pub... 2026-05_media/AHkbwyJDElPafZMJ8sTbC0W4IkbzaYDg2KXUCyW63R-9CA0A [05/27/2026 00:09] namibj Who with their "normal" mind wants "-0.32/-0.12/0.08 V (min/typ/max)" threshold voltage (`W/L= 10/1.8`; `VT0 (@Vds=0.1V,Vgs at Ids maximum Slope, Vt0=Vgs_intercept-0.5*Vds)`) after all.... [05/27/2026 00:12] namibj only good for cascodes and _I think_ speed-pushed MCML buffers. If one happens to have a negative supply they can also be used as high-side power gating n-channel switches, but turning them off seems to need a negative gate bias of around 500~1000 mV. [05/27/2026 00:12] mithro_ The details about mask layers that I have is at https://docs.google.com/spreadsheets/d/1RRPZ5prd2J3x1TO8l8eW8O50F2o9tF4JRuRT-NS01oU/edit?gid=2033140516#gid=2033140516 and https://docs.google.com/document/d/1Uf6xGZHpuA_G4iyAvxDwEENMjPBAf0Q16D0c8F0lbDQ/edit?tab=t.0 {Embed} https://docs.google.com/document/d/1Uf6xGZHpuA_G4iyAvxDwEENMjPBAf0Q16D0c8F0lbDQ/edit?tab=t.0 wafer.space - GF180MCU Cost Optimization Goal Figure out how to eliminate mask layers and reduce the cost of GF180MCU runs. Background The Google GF180MCU runs used a GF180MCU configuration called "1P5M26ML" which is a 5 metal, 26 mask layer configuration. This configuration costs almost 50% more for both masks and wafers than the "1P3... 2026-05_media/AHkbwyKWzUtTPSSgE4N7fL5Jr5N4gK-EN5iO0imgl0-55EF4 {Embed} https://docs.google.com/spreadsheets/d/1RRPZ5prd2J3x1TO8l8eW8O50F2o9tF4JRuRT-NS01oU/edit?gid=2033140516 wafer.space GF180MCU Configuration 2026-05_media/AHkbwyLupzKGcid_zXmJMikEm8OVvuGZrDvay277UH-1B1E7 [05/27/2026 00:19] namibj ahhh ok so is this mask 1L: `MVNVT`.... sadly https://gf180mcu-pdk.readthedocs.io/en/latest/physical_verification/design_manual/drm_04_2.html smells like NDA wall to me [05/27/2026 05:59] 246tnt AFAIK/U "native" device don't really need another mask. You need a mask to know where to put Vt implants but you need that mask anyway for normal transistors. The `NAT` GDS layer just tells the mask generation to not put implants on that transistor ... [05/27/2026 08:33] namibj They say the devices are optional that implies to me they're not truly free at just not under all cheapenings of the process. [05/27/2026 08:36] namibj And if native devices were free why do only thick oxide native devices exist on sky130A? They use the same GDS drawing layer as the low (but still very non-zero) threshold voltage `nfet_1v8_lvt`. [05/27/2026 12:15] rtimothyedwards_19428 @namibj : I have used native Vt devices pretty much exclusively for cascodes. The `05v0_nvt` device in sky130 works quite well as a cascode on top of a normal 3.3V nFET, and the gates of both devices can be driven with the same voltage, avoiding the need to set up a cascode voltage on the n side. [05/27/2026 12:17] namibj I'm looking to use them in especially upper tiers of an MCML pull-down-network. [05/29/2026 22:17] rtimothyedwards_19428 @namibj : Tomorrow's github update of magic (version 8.3.654) includes an overhaul of the way that magic determines whether or not to output a net as a detailed resistor network. The upshot is that it is now processing your quarter-sized chip in 20 minutes. However, it is calculating a completely different set of nets. Previously it was being forced to calculate resistor networks on all of the pins, regardless of whether or not the resistance is large enough to matter, and it missed a lot of internal nets because of the outdated way it was trying to determine the cutoff. It is also now much easier to control the number of nets being output and how much the resistor networks are simplified. With the default options, your chip, which has a total of around 300k nets, ran detailed resistance calculations on about 8% of those, and decided that around 6000, or about 2%, were large enough to be meaningful. A spot-check of the nets being output showed that the new method seems to work reasonably well. The resistor networks being output were for some very long wires in the digital (long enough to exceed 100 ohms in some cases), the row and column selection wires in the SRAM, and the wires running from the digital system outputs to the GPIO pads. The ground network is not extracted and I will implement a way to force specific nets like ground to be extracted when needed. ============================================================== Exported 213 message(s) ==============================================================