David Gessel
Cleaning up poor quality laser scans with Meshlab
Laser scanning often results in files that are massive, like 100s of megabytes, even gigabytes of data. This is rarely useful. Metrology and digitization of most manufactured parts should be done with touch probes for accuracy and clean data, 3 points define a plane not 300,000. One is reminded of the adage “a person with one clock knows the time, a person with two is never sure.” However, laser scanners scan fast and give cool looking results without much expertise, even if they make the data clean up slow. So here we are, 68 simple steps to a clean mesh.
MeshLab is an amazingly powerful tool that implements code straight out of current mathematical papers in python efficiently and well. It is a great tool, but unless you have time and the background to read all of the papers, it can be hard to choose which reconstruction or simplification or cleaning algorithm and what parameters will clean up a noisy mesh without turning it into a crazy starburst of near infinite polygons. This little post is a cookbook series of steps that reduced a 17m face cluster of 172 components and 15,934 boundary edges into a 902k face single component, water-tight, two-manifold, properly normalized, 3D printable mesh in “only” 3.5 hours of processing time (or about 6 hours with user inattention). It worked successfully 4 times on 4 different scans, so seems robust.
The basic process is to:
- Loop through some steps to clean up the import mesh, verifying with topological measures,
- Rebuild the mesh at high fidelity and resolution, making it huge but regular and consistent,
- Loop through the cleaning steps to make fix surviving crazies from the scan
- Decimate the huge mesh down to an appropriate uniform minimum feature resolution as needed to capture details, which will be too detailed in flat/simple areas,
- Clean up whatever crazies that process introduced,
- Use Quadric Edge Collapse Decimation to reach a target facet count/file size that can be ingested by downstream apps (typically <1m faces is a good target), which preserves edge detail intelligently.
- Clean up any crazies that algo created.
- Make the normals coherent
- Export the result. Yay.
This sequence reduced the input from 17M faces with severe topology issues to a clean, watertight 902K face mesh suitable for CAD import that preserved geometry and sufficient detail for reverse engineering, reproduction, and fit checks.
A detail of the original file looked like this, note the insane density of verticies:
After this clean-up sequence, the data is much less cluttered but critical details remain well defined and clear.
Note the edges still have plenty of points, but the relatively flat part are now encoded in a minimal number of points, sufficient relative to the feature sizes to represent geometry and surface quality.
I uploaded an annotated logfile of the process in case the below isn’t clear and the MeshLab .mlx script file with the filter parameters as well which can be kinda copy-pasted into the script window automate the process in blocks. If I have to do more of this, I’ll automated it further as a python script that would call PyMeshLab.py and conditionally fork on the results of the topological quality measure results. It’s a good way to keep your CPU busy.
Meshlab Sequence
Import the messy file into meshlab.
Initial State
- Starting mesh: 17,052,444 faces, 8,534,047 vertices
- 172 connected components, 15,934 boundary edges
- Non-manifold mesh with multiple topological issues
Phase A: Initial Cleanup and Repair
- Compute Topological Measures — 29.6 sec
- Identified initial mesh problems
- Merge Close Vertices — 27.8 sec
- Threshold: 0.1975 (0.01% of bounding box)
- Merged 166,446 vertices
- Remove Duplicate Faces — 5.3 sec
- Deleted 3,933 duplicate faces
- Remove Duplicate Vertices — 4.1 sec
- No duplicates found
- Remove Isolated Pieces — 14.5 sec
- Min diameter: 790.013 (40% of bounding box)
- Removed 136 of 137 components, 14,193 vertices
- Remove T-Vertices – 28.8 sec
- Method: Edge Collapse, Ratio: 40, Iterate: true
- Removed 1,745 T-vertices
- Remove Unreferenced Vertices — 0.2 sec
- None found
- Remove Zero Area Faces — 0.8 sec
- None found
- Select Non-Manifold Edges — 6.3 sec total
- Dilate Selection — 2.4 sec total
- Delete selected faces AND vertices
- Deleted 36,787 faces
- Repair Non-Manifold Vertices — 11.8 sec
- Method: Split vertices
- Split 42 non-manifold vertices
- Select/Delete Self-Intersecting Faces — 86.9 sec total
- Deleted 771 faces
- Close Holes — 52.5 min
- Max hole size: 300 edges, Refine: true
- Closed 1,598 holes, added 16,658 faces
At this point the model should be clean enough for surface reconstruction which will create a new mesh shrink-wrapped over the scanned mesh.
Phase B: Surface Reconstruction
- Screened Poisson Reconstruction — 12.6 min
- Depth: 14, Adaptive depth: 7
- Scale: 1.2, Samples per node: 1.5
- Interpolation weight: 12
This creates a new mesh layer – verify it is a good match to the old (toggle the “eye” for each layer on and off) and if it is good, close the eye of the original file, delete hidden layers, and then save as a new file name. This mesh is roughly 2x larger than the original and will be slow until it is decimated to a reasonable size.
Phase C: Clean up the new giant mesh
14-26. Repeat cleanup sequence (Phase A) — ~67 min total.
- Similar parameters as Phase A
- Removed 16 of 17 newly created disconnected components (floating glitches)
- Removed 32,758 T-vertices
- Removed 152 self-intersecting surfaces
- Select non-manifold, dilate, delete removed 6270 faces, 1661 vertices
- Closed 404 holes
- Assumed close enough for CD reduction (which is robust against topological oddities)
Phase D: Clustering Decimation
- Clustering Decimation — 14.9 sec
- Cell size: 0.988 (0.05% of bounding box) this means ~1 mm resolution on a part ~1.95 meters long, a parameter that is highly design dependent.
- Reduced to 4,486,934 faces
Phase E: Third Cleanup Round
28-40. Another cleanup cycle — ~49 min total.
- Removed 9,269 duplicate faces
- Deleted 128,584 non-manifold faces
- Closed 5,050 holes
Down to 4,406,986 faces, but 74 components so repeat the clean cycle:
41-52. Fourth cleanup iteration — ~48 min total.
- Removed 73 of 74 components
- Multiple hole closing operations
Single component, 4,405,554 faces, no holes, two manifold: OK for QECD.
Phase F: QECD Decimation
- Quadric Edge Collapse Decimation — 88.1 sec
- Target: 902,777 faces (20% of current)
- Preserve boundary: true, Planar simplification: true
- Planar weight: 0.001 (default)
Reduced to ~900K faces: this is an acceptable size for most programs to ingest.
Phase G: Final Cleanup
54-65. Final cleanup passes — ~8.5 min total.
Three iteration of cleanup sequence.
- Final topology check — 2.3 sec
- Final mesh: 451,053 vertices, 902,498 faces
- Single connected component
- Two-manifold, watertight (0 boundary edges)
- Genus: 99 (99 through-holes)
- Re-Orient Faces Coherently — 1.9 sec
- Export — 0.5 sec
Saved as an STL file, binary, no texture data (or as needed for next program)
Total Processing Time
Approximately 3.5 hours (mostly hole-closing operations which took ~2.5 hours total)
Disappearing checks in WordPress checkboxes
In some recent update, before 6.8.1 and still present in 6.8.2, WordPress changed the CSS for checkbox styling from checkbox to none
which, unshockingly, makes it awfully hard to know what a checkbox state is. Most wordpress sites will use the minified css file shipped with wordpress by default, which makes fixing the bug a little more tedious given browser inspectors prettify the minimized code making grepping for the fault a bit more challenging. After some digging around the bug is in the wp-admin/css/forms.css
file and the minified version wp-admin/css/forms.min.css
, and, of course, the RTL versions wp-admin/css/forms-rtl.css
and wp-admin/css/forms-rtl.min.css
.
The offending stanza is readable in the pretty versions as:
input[type="checkbox"], input[type="radio"] { border: 1px solid #8c8f94; border-radius: 4px; background: #fff; color: #50575e; clear: none; cursor: pointer; display: inline-block; line-height: 0; height: 1rem; margin: -.25rem .25rem 0 0; margin-bottom: 0px; outline: 0; padding: 0 !important; text-align: center; vertical-align: middle; width: 1rem; min-width: 1rem; -webkit-appearance: none; box-shadow: inset 0 1px 2px rgba(0,0,0,.1); transition: .05s border-color ease-in-out; }
-webkit-appearance: none;
should be -webkit-appearance: checkbox;
at least if you want to know the state of your checkboxes (you probably do).
A bit tedious to edit the minified versions but nano wp-admin/css/forms.min.css
followed by ESC->SHIFT-4 then searching for min-width:1rem;-webkit-appearance
and changing the following none
to checkbox
did the trick for me. This is the sort of thing that has already persisted for a few versions and may persist for more. Perhaps the devs have customized CSS so they don’t see the problem, so expect to make the change every update for a while.
Treegraph.sh a tool for generating pretty file structure graphs
Linux has a nice little tool in the repositories called tree
generates a nice console window-based directory map from the current location down. It’s great, but there’s not an equivalent in FreeBSD, though you can get the basic need met using find and some text processing pipes like
find . -type d -maxdepth 3 | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/"
. |-subdir0 | |-subsubdir1 | |-subsubdir0 |-subdir3 |-sub dir 2
However, this is enough to make me wish for something more visual. As usual, someone else already did it, but their version wasn’t quite what I wanted so I enlisted Claude.ai to help generate a little script that, like the above command, find
s it’s way through a directory structure and builds a .dot
file map as it goes, which can be converted into useful graphic formats with graphviz
.
The code (below or at gitlab) is pretty easy to use, it does the dotfile generation then just parse the dotfile like usual with graphviz, either using an installed version or online.
treegraph.sh -d /usr/home -f -s -m 2 > tree.dot dot -Tsvg tree.dot > tree.svg
The edges seem to overlap a bit more than I’d like, but that’s an issue with graphviz
and if you really want a presentation ready graph can be adjusted manually with invisible nodes or edges and manually set connection points to merge lines. It defaults to dot
, which makes sense for ranks, but looks cool with neato
or twopi
as well.
#!/usr/local/bin/bash # Help function Help() { echo "Usage: treegraph.sh [options]" echo echo "Options:" echo " -h Display this help message" echo " -f Enumerate both directories and files (default: directories only)" echo " -d DIR Specify the root directory to parse (default: current directory)" echo " -m DEPTH Specify the maximum depth of parsing (default: no limit)" echo " -s Compute the size of folders and files using 'du -sh' (default: off)" echo echo "Examples:" echo " treegraph.sh # Current directory, directories only" echo " treegraph.sh -d /usr/home -f # Parse /usr/home, include files" echo " treegraph.sh -d /usr/home -m 3 # Parse /usr/home with max depth 3" echo " treegraph.sh -s # Show sizes for current directory" echo " treegraph.sh -d /usr/home -f -s -m 2 # All options combined" echo echo "Example generating an SVG file (requires graphviz installed):" echo " treegraph.sh -d /usr/home -f -s -m 2 > tree.dot" echo " dot -Tsvg tree.dot > tree.svg" echo echo "When using -s, nodes are color-coded by size:" echo " B (bytes) - gray50" echo " K (KB) - black" echo " M (MB) - blue2" echo " G (GB) - darkorange3" echo " T (TB) - crimson" } # Initialize variables with defaults StartDir="." Type="d" MaxDepth="" MaxDepthArg="" ShowSizes=0 # Process options while getopts ":hfsd:m:" option; do case $option in h) # Display help Help exit;; f) # Include files Type="f";; s) # Show sizes ShowSizes=1;; d) # Directory StartDir="$OPTARG";; m) # Maximum depth MaxDepth="$OPTARG" MaxDepthArg="-maxdepth $MaxDepth";; \?) # Invalid option echo "Error: Invalid option" Help exit 1;; :) # Missing argument echo "Error: Option -$OPTARG requires an argument" Help exit 1;; esac done # Build find command based on Type if [ "$Type" = "f" ]; then TypeArg="" # No type filter for both files and directories else TypeArg="-type d" # Only directories (default) fi # Get absolute path of starting directory for proper path handling StartDirAbs=$(cd "$StartDir" 2>/dev/null && pwd) || StartDirAbs="$StartDir" # Extract base directory name for display BaseDir=$(basename "$StartDirAbs") # If showing sizes, get root directory size if [ $ShowSizes -eq 1 ]; then RootSizeInfo=$(du -sh "$StartDir" 2>/dev/null | awk '{print $1}') # Determine color based on size unit if [[ "$RootSizeInfo" =~ [0-9]+B ]]; then RootColor="gray50" elif [[ "$RootSizeInfo" =~ [0-9]+K ]]; then RootColor="black" elif [[ "$RootSizeInfo" =~ [0-9]+M ]]; then RootColor="blue2" elif [[ "$RootSizeInfo" =~ [0-9]+G ]]; then RootColor="darkorange3" elif [[ "$RootSizeInfo" =~ [0-9]+T ]]; then RootColor="crimson" else RootColor="black" # Default color fi SizeLabel=" ($RootSizeInfo)" ColorAttr=", fontcolor=\"$RootColor\"" else SizeLabel="" ColorAttr="" fi # Run find and generate graphviz dotfile ( echo "digraph directory_structure {" echo " node [shape=box,style=rounded,width=0.1,height=0.1,margin=\"0.07,0.01\"];" echo " edge [arrowhead=dot, arrowtail=dot, dir=both, arrowsize=0.3, color=gray];" echo " nodesep=0.15;" echo " rankdir=LR;" echo " concentrate=true;" echo " overlap=false;" echo " splines=true;" echo " searchsize=10000;" echo " \"$StartDir\" [label=\"$BaseDir$SizeLabel\"$ColorAttr];" # Process find output but skip the root directory itself find "$StartDir" $TypeArg $MaxDepthArg | grep -v "^$StartDir\$" | awk -v start_dir="$StartDir" -v type="$Type" -v show_sizes=$ShowSizes ' { current=$0; # Check if this is a file or directory is_file = 0; if (type == "f") { cmd = "test -f \"" current "\" && echo 1 || echo 0"; cmd | getline is_file; close(cmd); } # Get the parent path parent = current; sub(/\/[^\/]*$/, "", parent); # Remove last component for parent if (parent == "" || parent == ".") parent = start_dir; # Extract just the name of the current node (not the full path) split(current, path_parts, "/"); node_name = path_parts[length(path_parts)]; # Get size if requested size_info = ""; color_attr = ""; if (show_sizes == 1) { cmd = "du -sh \"" current "\" 2>/dev/null | awk '\''{print $1}'\''"; cmd | getline size_info; close(cmd); if (size_info != "") { # Determine color based on size unit if (size_info ~ /[0-9]+B/) { color = "gray50"; } else if (size_info ~ /[0-9]+K/) { color = "black"; } else if (size_info ~ /[0-9]+M/) { color = "blue2"; } else if (size_info ~ /[0-9]+G/) { color = "darkorange3"; } else if (size_info ~ /[0-9]+T/) { color = "crimson"; } else { color = "black"; # Default color } size_info = " (" size_info ")"; color_attr = ", fontcolor=\"" color "\""; } } # Create node with appropriate shape and just the name as label if (is_file) { printf " \"%s\" [shape=plain, width=0, margin=0, label=\"%s%s\"%s];\n", current, node_name, size_info, color_attr; } else { printf " \"%s\" [label=\"%s%s\"%s];\n", current, node_name, size_info, color_attr; } # Create node connection printf " \"%s\" -> \"%s\";\n", parent, current; } ' echo "}" ) # End of generated dotfile
FINAL System shutdown after 16 years
root@gamanjiru:/usr # shutdown -p now Shutdown NOW! shutdown: [pid 14442] root@gamanjiru:/usr # *** FINAL System shutdown message from gessel@gamanjiru.blackrosetech.com *** System going down IMMEDIATELY System shutdown time has arrived Connection to gamanjiru.blackrosetech.com closed by remote host. Connection to gamanjiru.blackrosetech.com closed. DISCONNECTED (PRESS <ENTER> TO RECONNECT) (Tue Feb 11 21:39:45 2025)
Almost 16 years ago, in July of 2009, I bought a used IBM x3655 7985-1AU off Ebay for $202.50 to replace my IBM Netfinity 5500 (4U!, dual dual 500MHz Xeo 1GB RAM (later upgraded to 4GB) and 5x 9.1GB HDs for $217.50 ) that had been running since 2004. That had, itself, replaced a dual socket generic deskside machine, a box Mark Godwin gave me back in the Nebucon days, that first went live on the interwebs running FreeBSD 2 in April of 1998 under black-rose.org. As of this post: 26 years, 10 months of FreeBSD hosted internets.
Those were the magic days of ebay: in 2008, just a year earlier, I’d quoted a similar x3650 (Intel E5410-based), 32GB but only a pair of crappy consumer SATA 500GB drives for $7,297.00.
The new x3655 came with 32GB of RAM and dual-core AMD processor and an RSA-II. The original motherboard firmware only supported dual core AMD processors, not the then brand new AMD Opteron quad core, so I bought a somewhat hard to find 43W7343 motherboard for $190 (and an additional 5 fans to max out cooling for $18 each) and then a pair of AMD Opteron 2352 2.1GHz CPUs and swapped the mobo and the CPUs and the heat sinks and the fans. Note that it is really hard to find data on the dual quad core option, it was a bit of a hot rod.
I added the 2x drive expansion module, another ebay find, and loaded the drive bays with 8x 26K5657 10k 72G 2.5″ SAS drives (at about $65 each, used) on the ServeRAID 8k and expanded the RAM to 64GB, 57856 MB available with RAM sparing set for reliability. The modified machine reports itself as an x3655 7943-AC1.
I ended up creating an 8×1 RAID array to pass to ZFS, ZRAID2, which mean I had a battery backed write cache I could count on. The system has dual power supplies, each connected to a 2200 VA (hacked XR) UPS.
Over the years, almost plural decades, of continuous operation I’ve lost maybe 4 drives. Once, while I wasn’t paying much attention, somewhere over about 6 months two drives failed and I got close to catastrophic failure, but ZFS pulled through. It started life on FreeBSD 7 and I took a chance on the then brand-new experimental release of ZFS. Over the years it ran most of the releases up to 12, stalling there as the decision was made to shift to new hardware and I’m pretty sure I’ve run every release of FreeBSD except 1.0 and 13.
It was a fast machine, 8⨉ 2100.12-MHz K8-class physical cores, more than enough to run mail and various web services, everything compiled for the AMD Barcelona architecture. The pre-Lenovoization of the IBM x86 hardware was really first rate, top of the line data center gear and it shutdown without a flaw, same performance and configuration as when I fired it up, used and already off-list, 16 years earlier.
It was never too slow, even now, even as OS’s have expanded and the total number of ports needed for basic services has grown by about 10x given ever spiraling dependencies. It wasn’t that it was slow, but that it used far too much power and electricity got more and more expensive in CA.
So I migrated to a new box, took the better part of a year to spare time migrate all of the jail services from the old machine and, increasingly unsupported FreeBSD 12 OS to new HPE DL360 G9 (ebay, inflation: $497) running FreeBSD 14, added a poudriere build environment on a DL60 and carefully tuned the kernels for power efficiency not bad for 10 disks, 20 cores, 192GB: 114W. Now there are 20 physical cores and 40 virtual and yet, clearly showing the limits of Moore’s law, the new box’s E5-2630 v4s are only 2.20GHz: 5x the execution cores, but only 5% faster clock. 16 years of progress.
Good night, sweet prince.
Adding a feature to MediaWiki WikiEditor formatting
MediaWiki is an excellent tool for maintaining documentation and I’ve had a self-hosted instance since at least 2011-06-04 (that’s the oldest edit in my internal user contributions list). And some 3,436 edits later, I still can’t remember the tags for SyntaxHighlight, which is an awfully nice little highlighter that uses pygments to render structured text in a more readable form. I got tired of looking them up every few weeks and so thought there must be a way to add some hints to the user interface.
I was surprised the WikiEditor plugin, which provides a nice point-n-click interface to some of the more commonly used MediaWiki markup tags, did not have an option or extension for SyntaxHighlight point-n-click and but, of course, you can edit the JavaScript that renders the toolbar and amend it with features you want.
The instructions are pretty clear, if not quite the step-by-step howto some are.
- Make sure you have WikiEditor enabled in LocalSettings.php
- You need permission to edit the Common.js page, which if you run the site you should have, but regular users can’t.
- If it doesn’t seem to load, make sure you clear all caches before testing.
On my site, the URL for Common.js
is https://your.host.tld/mediawiki/index.php?title=MediaWiki:Common.js
which contained only the default
/* Any JavaScript here will be loaded for all users on every page load. */
and to which I added:
/* Any JavaScript here will be loaded for all users on every page load. */
// Check if we're editing a page.
if ( [ 'edit', 'submit' ].indexOf( mw.config.get( 'wgAction' ) ) !== -1 ) {
// Add a hook handler.
mw.hook( 'wikiEditor.toolbarReady' ).add( function ( $textarea ) {
// Configure a new toolbar entry on the given $textarea jQuery object.
$textarea.wikiEditor( 'addToToolbar', {
section: 'advanced',
group: 'format',
groups: {
list: {
tools: {
syntaxhighlight : {
label: 'SyntaxHighlight',
type: 'select',
list: {
'bash': {
label: 'Bash',
action: {
type: 'encapsulate',
options: {
pre: '<syntaxhighlight lang="bash">',
post: '</syntaxhighlight>'
}
}
},
'unixconfig': {
label: 'Config',
action: {
type: 'encapsulate',
options: {
pre: '<syntaxhighlight lang="unixconfig">',
post: '</syntaxhighlight>'
}
}
},
'apacheconf': {
label: 'ApacheConfig',
action: {
type: 'encapsulate',
options: {
pre: '<syntaxhighlight lang="apacheconf">',
post: '</syntaxhighlight>'
}
}
},
'json': {
label: 'JSON',
action: {
type: 'encapsulate',
options: {
pre: '<syntaxhighlight lang="json">',
post: '</syntaxhighlight>'
}
}
},
'patch': {
label: 'Patch',
action: {
type: 'encapsulate',
options: {
pre: '<syntaxhighlight lang="diff">',
post: '</syntaxhighlight>'
}
}
},
'php': {
label: 'PHP',
action: {
type: 'encapsulate',
options: {
pre: '<syntaxhighlight lang="php">',
post: '</syntaxhighlight>'
}
}
},
'javascript': {
label: 'JavaScript',
action: {
type: 'encapsulate',
options: {
pre: '<syntaxhighlight lang="javascript">',
post: '</syntaxhighlight>'
}
}
},
'html': {
label: 'HTML',
action: {
type: 'encapsulate',
options: {
pre: '<syntaxhighlight lang="html">',
post: '</syntaxhighlight>'
}
}
},
'css': {
label: 'CSS',
action: {
type: 'encapsulate',
options: {
pre: '<syntaxhighlight lang="css">',
post: '</syntaxhighlight>'
}
}
},
'arduino': {
label: 'Arduino',
action: {
type: 'encapsulate',
options: {
pre: '<syntaxhighlight lang="arduino">',
post: '</syntaxhighlight>'
}
}
},
'perl': {
label: 'Perl',
action: {
type: 'encapsulate',
options: {
pre: '<syntaxhighlight lang="perl">',
post: '</syntaxhighlight>'
}
}
},
'python': {
label: 'Python',
action: {
type: 'encapsulate',
options: {
pre: '<syntaxhighlight lang="python">',
post: '</syntaxhighlight>'
}
}
}
}
}
}
}
}
} );
});
}
and coolio, it works:
Optane, a modern technology tragedy (plus FreeBSD nvmecontrol)
Intel won the storage wars. They invented a storage technology in 2015 that was the best of everything: almost as fast as (then) RAM, basically infinite write endurance in any normal use, and fairly cheap. They even made a brilliant config on m.2 with integrated supercap for power-failure write flush. Just awesome and absolutely the write tech for modern file systems like ZFS. It is perfect for SLOGs. You wish you had a laptop that booted off an Optane m.2 You wish your desktop drives were all NVME Optane.
Well, wishes are all we got left, sadly. Optane, RIP 2022.
You can still buy optane parts on the secondary markets and it seems some of the enterprise DC products are at least still marked current on Intel’s website, but all retail stocks seem to be gone.
Man was that an amazing deal at $0.50/GB. In my application, the only practical form factor was M.2 and even that was a bit wonky in an HP DL360 G9, but more on that later. There are a variety of options and most are available on the used market:
PN | Intro | Cap GB | Write MB/s | write k iops | PBW endurace | PLP | $ (market, 2024) |
MEMPEK1W016GAXT | Q1’17 | 16 | 145 | 35 | 0.2 | NO | 5 |
SSDPEL1K100GA | Q1’19 | 100 | 1,000 | 250 | 10.9 | YES | 109 |
SSDPEL1K200GA01 | Q1’19 | 200 | 2,000 | 400 | 21.9 | YES | 275 |
SSDPEL1K375GA | Q1’19 | 375 | 2,200 | 550 | 41 | YES | 800/1,333/NA |
SSDPEK1A058GA | Q2’22 | 58 | 890 | 224 | 635 | YES | 32/140 |
SSDPEK1A118GA01 | Q2’22 | 118 | 1050 | 243 | 1292 | YES | 70/229 |
Any of these would be a good choice for a SLOG on rotating media, but the later ones are just insane in terms of performance, and that’s compared to enterprise SSDs. They pricing cratered after they were canceled and dangit, didn’t get em. The used market has gone way up, better price increase than bitcoin over the same period and they’re not virtual beanie babies! The SSDPEL1K100GA is the best deal at the moment and has a beefy supercap for power continuity and is still $818 on Amazon, apparently introduced at $1,170. This pricing might have explained why Optane didn’t do better. The 375 GB M.2 would be an awfully nice find at $0.50/GB, that’d be a pretty solid laptop boot disk.
Hardware
For SLOG you really want two devices mirrored in case one fails. The risk of an optane DC grade device failing is trivial and given it has Power Loss Protection, the most likely cause of failure and why your main array failed to write out the transactions committed to the SLOG, we’re really talking about media failure and as it is 3D X-Point it is NOT going to wear out like NAND, it’s rational to single-disk it. I almost striped mine but in the end decided against it because that quadruples the fail rate over a single device and 8x over mirrored and I don’t really need the space.
So how do you install two M.2 devices in a computer that doesn’t have M.2 slots on the mobo? With a PCI card, of course. But wait, you want two in a slot, right? And these are x4 devices, the slots are x8 or x16, so two should be able to pair, right?
Not so fast. Welcome to the bizarre world of PCI furcation. If you want to add two drives to the core PCI bus, you have to split the bus to address the cards. Some mobos support this and others do not. As shipped, the HPE DL360 G9 did not.
BUT, a firmware update, v 1.60 (April 2016) added “support to configure the system to bifurcate PCIe Slot 1 on the DL360 Gen9 or PCIe Slot 2 on the DL380 Gen9.” W00t. A simple Supermicro AOC-SLG3-2M2 supports 2x M.2 cards and only requires bifurcation to work, all good.
Not so fast. In order to pack the DL360 G9 with 2.5 SSDs, you need a Smart Array Controller (set for passthru for ZFS) and that sits in slot 1 and while I believe it can go in any X16 slot, the cabling is not compatible and that’s a lotta SAS cables to replace. Bifurcation on the mobo is out.
But you can fucate on a PCI card just as well – likely this adds some latency and it’d be interest to perf test against more direct connections. I ended up choosing a RIITOP dual M.2×22110 PCI card and it worked out of the box transparently, both disks showed and while I’m not getting 250,000 IOPS, performance is good. It is based on the ASMedia ASM2812, seems like a reasonable chip used in a lot of the lower cost devices of this type, most with 4x M.2 slots instead of 2.
Software
FreeBSD recognizes the devices and addresses them with nvmecontrol
. You can pull a full status report with, for example nvmecontrol identify nvme0
, which provides information on the device or nvmecontrol identify nvme0ns1
which gives details about the storage configuration, including something important (foreshadowing) the LBA format (probably #00, 512).
Current LBA Format: LBA Format #00 ... LBA Format #00: Data Size: 512 Metadata Size: 0 Performance: Good LBA Format #01: Data Size: 512 Metadata Size: 8 Performance: Good LBA Format #02: Data Size: 512 Metadata Size: 16 Performance: Good LBA Format #03: Data Size: 4096 Metadata Size: 0 Performance: Best LBA Format #04: Data Size: 4096 Metadata Size: 8 Performance: Best LBA Format #05: Data Size: 4096 Metadata Size: 64 Performance: Best LBA Format #06: Data Size: 4096 Metadata Size: 128 Performance: Best
The first thing I’d do with a used device is wipe it:
gpart destroy -F /dev/nvme0 gpart destroy -F /dev/nvme1
I would not bother formatting the device to LBA 03/4k. Everyone tells you you should, but you don’t get much of a performance increase and it is a huge pain because nvmecontrol
currently times out after 60 seconds (at least until the patch needed is pushed to kernel or you recompile your kernel with some fixes) if you did want to try, you’d run:
# time nvmecontrol format -f 3 -m 0 -p 0 -l 0 nvme0 316.68 real 0.00 user 0.00 sys (no errors)
-f 3
sets LBA Format #03, 4096 which should give “Performance: Best
” which certainly sounds better than “Good
.”
But it’ll error out. You need to mod /usr/src/sys/dev/nvme/nvme_private.h
with the below modifications and recompile the kernel so it won’t time out after 60 seconds.
#define NVME_ADMIN_TIMEOUT_PERIOD (600) /* in seconds def 60 */ #define NVME_DEFAULT_TIMEOUT_PERIOD (600) /* in seconds def 30 */ #define NVME_MIN_TIMEOUT_PERIOD (5) #define NVME_MAX_TIMEOUT_PERIOD (600) /* in seconds def 120 */
Performance Aside
I tested 512 vs 4k in my system – and perhaps the AIC’s bridge latency or the whole system’s performance so limited the performance of the optane cards that a no difference would appear, these cards do rock at the hardware level (this is with 4k formatting):
# nvmecontrol perftest -n 32 -o read -s 4096 -t 30 nvme0ns1 && nvmecontrol perftest -n 32 -o write -s 4096 -t 30 nvme0ns1 Threads: 32 Size: 4096 READ Time: 30 IO/s: 598310 MB/s: 2337 Threads: 32 Size: 4096 WRITE Time: 30 IO/s: 254541 MB/s: 994
That’s pretty darn close to what’s on the label.
However, testing 512 vs. 4k formatting at the OS level (didn’t test raw) it was a less extraordinary story:
LBA/FW ver. | 4k E2010650 | 512 E2010650 | 4k E2010485 | 512 E2010600 |
Median Mb/s | 759.20 | 762.30 | 757.50 | 742.80 |
Average Mb/s | 721.70 | 722.87 | 721.64 | 724.35 |
Definitely not +10%
So I wouldn’t bother reformatting them myself. Testing a few configurations with
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1
I get
Device\Metrics | Max IOPS | Avg WBW MiB/s | avg SLAT µS | avg LAT µS |
10 SAS SSD ZFS Z2 Array | 20,442 | 1,135 | 4,392 | 53.94 |
Optane 100G M.2 Mirror | 20,774 | 624 | 3,821 | 95.77 |
tmpfs RAM disk | 23,202 | 1,465 | 6.67 | 42 |
Optane is performing pretty close to the system limit by most metrics – the SLAT and LAT metrics are highly dependent on software.
Formatting
I did something a bit funky since 100GB is way more than this little server could ever use for SLOG. I set it at 16GB which is probably 4x overkill, then used the rest as /var mountpoints for my jails because the optanes have basically infinite write endurance and the log files in var get the most writes on the system. I’m not going into much detail on this because it’s my own weird thing and chances anyone else cares is pretty small.
Initialize GPT
gpart create -s gpt nda0 gpart create -s gpt nda1
Create Partitions
gpart add -b 2048 -s 16g -t freebsd-zfs -a 4k -l slog0 nda0 gpart add -b 2048 -s 16g -t freebsd-zfs -a 4k -l slog1 nda1 gpart add -s 74g -t freebsd-zfs -a 4k -l ovar0 nda0 gpart add -s 74g -t freebsd-zfs -a 4k -l ovar1 nda1
ZPool Operations
zpool add zroot log mirror nda0p1 nda1p1 zpool create optavar mirror nda0p2 nda1p2 zpool set autotrim=on optavar
Create Datasets
zfs create -o mountpoint=/usr/local/jails/containers/jail/var -o compression=on -o exec=off -o atime=off -o setuid=off optavar/jail-var etc
Electronic Signatures and PDF
Electronic signatures are a technology that has been bizarrely slow to mature. Lots of documents still rely on the idiotic premise that some stupid graphic somehow serves as a secure measure of document authenticity. This might have had some slight measure of validity in the days of actual paper documents being required with “wet signatures.” but the premise of face-to-face document signing ceremonies should have long been consigned to history with signet rings, let alone a global transit trade in random bits of paper bearing binding proof of commitment.
First the Uniform Electronic Transactions Act (UETA, 1999) then H.R.1714/S.761, Electronic Signatures In Global and National commerce (E-Sign) act (2000) was signed (ha) into law (probably with a wet signature), now Public Law 106–229, it has been legally binding to sign documents with electronic signatures for 25 years.
So why is it almost never done? Why do are we still sometimes asked to fax “signed” documents?
Why do we fax “signed” documents? Because lawyers and legislators are unbelievably, almost incomprehensibly ignorant of the most basic operational functions of technology and absolutely too stupid, too utterly moronic, mindbogglingly dense and incomprehensibly dumb that… and I am NOT making this up… but seriously… there are people who actually have an impact on laws and legal matters who believe that fax transmissions are more “secure” and less prone to interception, manipulation, or hacking than email. Yes, people who believe this kind of thing are actually allowed to practice law. Truly tragic but still true. The world suffers that such profound ignorance persists.
Have you ever tried to electronically sign a document? Turns out it isn’t trivial and the burden isn’t the core technology or concept but a few problematic implementation steps.
The first barrier is the interjection of the certificate mafia’s profit motives. Various corporate monsters saw an opportunity to make bank exploiting the aforementioned abject technical ignorance and utter technical incompetence of our legislative and legal infrastructure and build a certification model that relies on pay-for-validation, lying that this would somehow ensure authenticity and people were too dumb to question the obvious idiocy of this stupid model. Even today, we rely on the good graces of the Mozilla foundation’s Let’s Encrypt to make secure communication viable because various OS and browser level dumbness considers self-signed certificates insecure for the stupidest, most reprehensible reasons possible. But Let’s Encrypt, bless them, won’t give you an X.509 signing certificate.
We’re all lucky CACert.org steps into this horrific void and, while it is complicated, offers an extremely secure, highly reliable, and (most importantly) free process for getting yourself an X.509 signing certificate. In order to get a signing certificate, you have to validate your identity in person at a meet up using their points system, a process that is infinitely more secure than any of the for-profit signing certificate providers that consider willingness to pay proof of identity. The USG should offer X.509 client certificates for free with passports and RealID renewals, but I’d still use CACert myself, cause they’re awesome.
For now: first, set up an account on CACert, install their root certificates in your OS and browser (why aren’t they included by default? Ask the certificate mafia.) You’ll need to do one of the things they require to prove you are who you claim (yes, actual security, unlike ANY of the commercial certificate providers, unreal how insanely stupid this process is) and then have CACert issue a Client Certificate.
Assuming you have your points with CACert, the basic process is fairly well documented:
You need to generate a signing request in your name, which you can do with OpenSSL, but it is easier using CACert’s nice online process.
It will take a few seconds (60?) and I’m not sure about the compatibility problems that might arise from a longer key, there are some bugbears once we try to use lamo corporate commercial software, but 4096 worked for me. You MUST GET YOUR PRIVATE KEY and save it to your OpenSSL enabled computer.
Remember to press the red “Show private key” button and copy/save the private key to a secure directory on your computer, you’re gonna need it later to convert the certificate into something dumb ass spyware Windows computers can use, which you need because Acrobat forms still can’t be signed without Adobe’s awful spyware Acrobat Reader.
(note the actual private key has quite a bit of text between the Begin and End lines but you know… redacted for privacy). Then click the blue “Copy CSR to Clipboard” button and switch over to New Client Certificate window and paste it where you’re supposed to.
You need the .crt version of the certificate to continue and that private key text file you saved earlier for the next step, as well as downloading the CACert root certificate and then you need openssl working (should be on most real computers, Windows or Apple is beyond my interest) and merely execute this one simple command:
$ openssl pkcs12 -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -nomac -export -out DavidGessel_3d_sha1.pfx -inkey private_key_for_CAcert_CSR.txt -in gessel@blackrosetech.com.crt -certfile CA_Cert_root_X0F.crt
To explain:
- openssl will generate a combined binary version of your certificate in pkcs12 format
- because Windows and Acrobat suck, you have to specify moderately insecure crypto: SHA1-3DES rather than the Linux default of AES 256 because why would a monopoly company like Microsoft have any incentive to fix bugs? If you don’t Windows will tell you “The password you entered is incorrect” to unlock your key because why fix bugs when corporate IT types are just utterly incompetent and will only specify windows no matter how awful and unusable it is because point-n-click?
- -nomac is another setting Windows needs to be able to use the cert and if you don’t specify this Windows will tell you “The password you entered is incorrect” again, because Windows does not care if it works for you because you have no choice.
- The -out certificate is what’s being generated and Windows native is .pfx, but .p12 will work too.
- The -inkey is the private key you remembered to save using the red button before (right? you need that).
- the -in (file) is the Client Certificate in normal X.509 .crt format real computers understand that CACert generated for you.
- the -certfile is CACert’s root certificate.
Now, WØØt, you have a certificate that should work. Go over to your dumb Windows machine and make sure you import the CACert root certificates – you just download them and then right click and select “install certificate” for the class 1, then the class 3, then the .pfx certificate you just created.
Now, finally, you can sign a document like someone who actually uses a computer rather than a quill and parchment to process documents.
Acrobat is another program that just doesn’t care too much about usability or user experience, so different versions might work differently. I had to click the “Signature Panel” button to open a sidebar to show the signature fields then right click and then choose my sig and click sign and save.
One final note about the state of signing in FOSS: it kinda sucks still. Various entities that use acrobat fairly well will generate forms with standard signature locations which you can print and sign and fax (not email) like we’re still waiting for Y2K or print and sign and snail mail if we are nostalgic for the pre-telephone era, or click and sign and email like we’re in the 21st century.
I’m not aware of any FOSS program that handles signature fields in the expected way. You can sign a whole pdf document with a variety of FOSS tools, and CACert has a good summary of these, but that signature, while binding on the document as a whole does not show in the form fields and so whatever non-tech functionary is asking you to sign the document is never going to understand how your e-sign compliant signature is binding and is going to insist you take a time machine back to the mid-80s to find a working fax machine unless you use Acrobat, which means Windows or Mac at least in a VM. You might be able to get some version of Acrobat to work in Wine, but you’ll need an old one that uses an internal certificate store rather than relying on the windows version (pre Acrobat X, I’m pretty sure).
Fun, huh? Basic digital functions are still broken decades after introduction but we have AI generated Teledep influencers telling us doubleplus buy useless beauty products and trust their health and exercise advice.