Skip to content

Fix benchmark for fast functions - #620

Merged
ai merged 4 commits into
ai:mainfrom
orhanayd:fix-benchmark
Oct 16, 2026
Merged

ai merged 4 commits into
ai:mainfrom
orhanayd:fix-benchmark

Conversation

@orhanayd

@orhanayd orhanayd commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Closes #619

tinybench times every call on its own. For fast functions, reading the timer takes longer than the function, so the results stopped near the same number (about 20M ops/sec in the README).

Now every task calls the function 1000 times in a loop, and the result is multiplied by 1000. The results are kept in an array, so V8 cannot skip creating the IDs.

Results on my machine (Apple M2, Node 22.22.2):

                        before          after
nope-id             30,798,652     92,171,643
nanoid              23,301,979     60,985,638
customAlphabet      15,149,795     48,901,502
crypto.randomUUID   16,576,011      7,209,005
uuid v4             14,139,569      6,669,449
uid                 28,466,420     56,244,592

crypto.randomUUID and uuid v4 go down. I checked both with a plain loop (one timer for 2M calls) and got similar numbers (crypto.randomUUID about 8.7M, uuid v4 about 7.8M ops/sec), so the old method showed them too high. Their numbers change a bit between runs (about 6.5M to 9M).

I did not change the README numbers. The Benchmark workflow runs on this PR, so the new numbers will be in its log.

Checked: pnpm test passes, and oxfmt --check test/benchmark.js is clean.

@ai

ai commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

New results from CI (Node.js 26, Docker image in some low-level VPS):

nope-id                 59,034,753 ops/sec
nanoid                  38,883,215 ops/sec
customAlphabet          25,481,101 ops/sec
crypto.randomUUID        6,871,047 ops/sec
uuid v4                  6,169,740 ops/sec
@napi-rs/uuid            8,791,173 ops/sec
uid/secure               8,229,214 ops/sec
@lukeed/uuid             6,687,021 ops/sec
nanoid for browser         653,420 ops/sec
secure-random-string       531,531 ops/sec
uid-safe.sync              534,386 ops/sec
Non-secure:
uid                     36,726,989 ops/sec
nanoid/non-secure        2,753,530 ops/sec
rndm                     2,820,063 ops/sec

Is it ok for you?

@ai

ai commented Oct 15, 2026

Copy link
Copy Markdown
Owner

Can you also put numbers from CI to README.*.md?

@orhanayd

Copy link
Copy Markdown
Contributor Author

Can you also put numbers from CI to README.*.md?

Yes, looks good to me. I added the CI numbers to all README.*.md files.
One more thing: all README files say "50% faster than native crypto.randomUUID()". With the new numbers nanoid is about 5.7 times faster (38.9M vs 6.9M ops/sec). I did not change this line. Should I update it?

@ai

ai commented Oct 15, 2026

Copy link
Copy Markdown
Owner

Let's replace with just faster than native crypto.randomUUID() :D The numbers are too big, nobody will believe.

Thanks

@orhanayd

Copy link
Copy Markdown
Contributor Author

Done, all README files now say "Faster than native crypto.randomUUID()".

I also found a mistake in my benchmark change, sorry about that. I used throughput.mean from tinybench. It is the average of the speed of each sample, and it shows too high numbers when some samples are much slower than others. For example, customAlphabet refills its pool from time to time. Samples with a refill are slow, samples without it are very fast, and the average of the speeds counts the fast ones too much.

Now the script divides the total number of calls by the total time: (N * 1000) / task.result.latency.mean. I checked all 14 tasks against a plain loop on my machine (a fresh process for each function, one timer for millions of calls). All of them are within about 6%, except the UUID tasks, which change more between runs (about 6.5M to 8.4M for crypto.randomUUID). Some examples:

                    old formula   new formula   plain loop
nope-id                    ~95M          ~74M         ~75M
nanoid                     ~62M          ~34M         ~34M
customAlphabet             ~47M         ~5.2M        ~5.3M
crypto.randomUUID         ~8.9M         ~7.8M        ~8.1M

So the README numbers from the last CI run are too high, mostly for nanoid and customAlphabet. When the new CI run is done, I will update the README numbers with its results. Please do not merge before that.

@ai
ai merged commit 779ceb7 into ai:main Oct 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark: results near 20M ops/sec are limited by timer overhead

2 participants