Climbing the Hill of Experiments

Published on March 29, 2025 8:37 PM GMT

A better anything can be achieved with simple tests at low costs.

Background

People often settle for "good enough" and "if it ain't broke don't fix it" in their personal lives, opting not to make any effort to improve said things because either:

Time, money, and/or effort can be spent elsewhere for a higher expected valueThey think it can't be improved

But how is one to tell how much better something can get or if it's already optimal? The only answer is to experiment. Most people have significant room for pareto improvements in their lives. The impact and availability of said improvements varies from low to high depending on the cost one is willing to incur and how much has already been attempted or implemented.

Costs, or Lack Thereof

Experimentation is often associated with major costs. Setting up experiments and collecting and analyzing data takes a lot of time. Thinking of all the controls and confounders takes a mental toll. Purchasing supplements or technology costs money. These are all in addition to the corresponding opportunity costs. "If experiment X doesn't pan out, I could've been doing Y all along, which I know brings me value" is a fair, common criticism against potential tests.

But experiments do not need to be so costly. Erring on the side of lower cost is key to ensuring experiments keep running; too high of a cost in any area (time, effort, money) will make experiments less likely to happen in the future. Design of experiments (DOE) has its place for areas that have high potential returns, while a simple "do X for Y" (e.g., take magnesium before bed for 30 days) and see how you feel has its place for lower returns or lower interest. The latter type is where I think a majority of benefits lie because they are more likely to be performed, there is a greater number available to test, and they are straightforward to implement.

These simple experiments are akin to hill climbing, defined by Wikipedia as:

an iterative algorithm that starts with an arbitrary solution to a problem, then attempts to find a better solution by making an incremental change to the solution. If the change produces a better solution, another incremental change is made to the new solution, and so on until no further improvements can be found.

The beauty and strength lie in the fact that the solution doesn't have to be arbitrary—it can be reasonable and informed, expediting the search time for the best solution and increasing the rate of improvements across the board. Further, improvements to multiple problems can be pursued at any given time without major interference with one another. This is one reason advice, especially pieces of such that are reliably backed, is so valuable: it is easy to implement, easy to verify the effectiveness, and can be backed out of quickly. Quick feedback loops lead to quick improvements and quick improvements lead to more testing.

I suspect the cost type that is most important to someone is the one they have the least of (e.g., if someone has lots of money and energy, but little time, they're time poor). This should be recognized, accepted, and accounted for when planning experiments. In other words, figure out your type of poorness, accept it, then find ways to avoid said cost in experiments and leverage the rich types.

A few notes on individual cost types:

Time

Time can be saved by outsourcing both physical and mental labor. Trying to see the effect of a clean house on happiness? Pay someone to do it. Trying to analyze data? Get an LLM to help with it.

Experiments also don't need to take an hour of planning, an hour of executing, and another hour of analysis to see if it actually worked. (Sure, the scientist in you may be loudly protesting about placebos and the need for controls in certain experiments, but sometimes just feeling better or doing better is enough for it to be considered effective.)

Effort

Effort, while often intertwined with time, is still distinct: some tasks can be short and tedious, long and mundane, or somewhere between the two. Again, effort can be reduced or almost altogether eliminated by outsourcing labor with a focus on making tasks easy and simple.

Effort is often inversely related to enjoyment, so experiments that are more fun will feel less effortful than if they were soul-sucking.

Money

Running cost-benefit analyses is helpful to determine if the experiment is worth running. Items that didn't work out can be sold on public marketplaces to recoup some of the cost. Ask others if they're willing to subsidize the cost in exchange for well-organized and well-planned results.

Diminishing Returns

Diminishing returns exist across all cost types, whether it's putting in more time, more effort, or more money. Try to recognize when returns plateau and move to the next experiment when/if that happens.

Getting Started

Step 1: Brainstorming

First, a list of potential experiments should be made from the following methods:

Examples: Improving poor sleep through supplementation or sleep hygiene practices; not eating healthily because of a poor meal prep routine; not exercising because of inconveniences that act as barriers.

self-blinded L-theanine RCT

QS: Sleep

generic QS

SLIME MOLD TIME MOLD

list on other online scientists

Experiments on a $50 DIY air purifier you can make in 30s

Examples: Dream journaling, magnesium, and melatonin for lucid dreaming; styles, consistencies, and removing barrier to entry for exercise

Step 2: Prioritization

Second, prioritize experiments based on expected return over time, or area under the enjoyment-time curve. The formula I use to think about this is:

priority = success-probability × value-per-time ÷ how-long-it-takes-to-implement

where the scales are 0-1 for success-probability, 0-10 for value-per-time, and 0-10 for how-long-it-takes-to-implement.

For example, magnesium supplementation may be 0.8 × 5 × 1 = 4 and consistent bedtime is 0.9 × 10 × 1/5 = 1.8. In other words, don't delay the magnesium until after the consistent bedtimes, but rather take care of the magnesium now while still starting the bedtime.

Probabilities can be estimated from literature (preferable), other n=1 experimenters or trusted figures (a bit less preferable), or raw (least preferable). Value per time is entirely subjective, but should be easily approximated. Implementation time depends on the depth of DOE—something like controversial supplementation may take longer to prove its value while increasing lighting brightness inside the home may have an immediate, noticeable effect.

Step 3: Planning

Third, plan exactly how to implement the experiment. Like estimating probabilities, literature or articles/blogs/podcasts/word-of-mouth can be good starting points for both design and execution.

Planning should include the following:

Which products, if any, you'll use. Search internet forums and parse reviews for the best while still taking into account personal type-poorness.How you'll track effectiveness. Vibes, metrics on pen-and-paper/phone/laptop/special software, other people's observations, raw output?A quantifiable quitting point if it doesn't seem to be working. No need to spin wheels when there are other opportunities.An actual procedure for how to administer the experiment, including mapping out all the options to test. This can be as simple as "take 200 mg magnesium before bed" to more complex structures that control for other variables.

Step 4: Performing

Fourth, do it. Purchase the products, set up the effectiveness tracker, define the quitting point, and follow the procedure.

Examples

Here's a non-exhaustive, vaguely-categorized list of as many experiments as I could think of in a few hours. Again, some of these are simple one-time behavior modifications that may reap surprising benefits, while others are long-term systems that must be maintained. (I reserve the right to not update regularly, but will try to as new ones come to light.)

music vs. no music

Takeaways

Doing something sub-optimal is often better than delaying or never doing the optimal.

There is almost always room to improve something at a low cost.

Speed matters. Get experiments done quickly so the "cost of doing something new will seem lower in your mind [and] you'll be inclined to do more".