Yes, it does get lower.

The effect you see is called internal resistance:

A practical electrical power source which is a linear electric circuit may <...> be represented as an ideal voltage source in series with an impedance. This impedance is termed the internal resistance of the source.

Put simply, a battery is not an ideal voltage source. A typical battery (i. e. non-ideal voltage source) will look like this:

What you are measuring is voltage between terminals A and B. According to Ohm's Law:

$$ U_{AB} = E * frac{R}{R+r} $$

• When there is no circuit, you can imagine your voltmeter's internal series resistance $ R_{volt} $ taking the role of $ R $. However, $ R_{volt} $ is usually so large (tens or hundreds of megaohms) compared to $ r $ (usually fractions of an ohm) that $ frac{R_{volt}}{R_{volt}+r} $ tends to 1, hence measured open-circuit voltage tends to the battery's internal (true) voltage $ E $.




• When there is a closed circuit with equivalent series resistance of $ R $, you will be able to see that the measured voltage $ U_{AB} $ drops proportionally to $ R $, in accordance with the above formula.

So, the voltage drop is real — the measured voltage is what your load gets. The more current it draws from the battery, the lower is voltage it gets.

When the battery is open you are measuring an open cell voltage. When the battery is in the system it's closed cell voltage under load. You are dropping some voltage across the internal impedance of the battery because your system is drawing current when the measurement is being made (so at the terminals the voltage is indeed lower). So both measurements MCU and multimeter are correct, the difference is that the multimeter is >1Mohm load while the MCU is much lower (since probably drawing at least mAs of power).

There may be another effect at play. Batteries do exhibit a recovery phenomenon where if left open cell with no load some of the voltage will recover after a time interval.

Every battery has a certain amount of output resistance. What happens if current flows through a resistor? Yes, a voltage drop! So the more current you draw from the battery, the lower the output voltage is.

This is true of all power supplies

Indeed, batteries sag their voltage on being loaded. So does everything else.

The main culprit is Ohm's Law, E=IR, where voltage drop across any conductor is proportional to its amperage drawn.

Part of a battery's sag is chemical, but part is simply the Ohm's Law resistance of its internal components.

Let's suppose you have a mad gaming rig with 4 paralleled video cards, the combo pulls 1000 watts when gaming. But it's just sitting at the Windows Homescreen and only pulling 100 watts. The power cables are carrrying 20A@5V, and dropping 0.01 volts, so the cards get 4.99 volts. (The wires are 2000 Siemens == 1/2000 Ohms.)

At this light load, the AC power supply is inefficient and poor power factor, so it is drawing 240VA or 2 amps off the 120V mains. The branch circuit wiring back to the panel is dropping 0.4 volts. Conductance is 5 Siemens == 1/5 ohm.

Now you fire up your most demanding game. Pulling 200A at 5V, the resistive losses alone inside your PC's wiring jumps to 0.1 volts. So the cards get 4.90 volts. That's a drop.

Meanwhile, the power supply draws 10A (1200VA) from the AC mains. Wiring voltage drop predictably increases to 2.0 volts, so voltage at the power suppy is 118V. Most likely a switching power supply pulls a skitch more current to compensate, otherwise its output voltage would sag also.

No current is being drawn on safety ground, so it's not dropping. Measured from ground, neutral is 1 volt and hot is 119 volts. And we can use this to affirm correct wiring. It's like the pointer bar on a torque wrench, it doesn't bend.

Of course, similar drops are happening all the way back to the power plant. There, increased load (in amps) sags the voltage because of the internal resistance of the generator, but also due to turbine horsepower. VA=W. If A increases beyond spec, V must decrease in proportion so W can remain within the turbine's ability. Having the turbine bog and slow down is not an option, because it's AC power and must stay in sync.

All batteries have a memory effect when unloaded such that they return slowly to near the previous voltage after a short burst load. There is also momentary quick drop in voltage due to a load of ESR*I = Δ V.

So both measurements must be taken at the same time to check calibration for errors and consider the amount of hysteresis thresholds required to prevent oscillation of sleep, wake-up cycles.

The memory effect time constant can be several to many minutes depending on the "no-load" leakage current after a load.

Because of these combined effects which might be computed for a given cell (ΔV= ESR * V/Rload + t/ESR * C2) the cut-off voltage is often lowered to capture the charge stored in memory capacitance C2 as long you know it returns to the safe Vmin threshold. Battery rapid aging occurs for the amount of time below its Vmin threshold.

Review the battery datasheet for details.

There is drop in voltage due internal resistance of the battery coming into play so you will see the voltage drop by a value of i * r ( where i is the current flowing and r is the internal resistance of the battery)

A new battery will have much less loaded voltage drop than you have.
An old, worn out, or damaged Lithium battery has a much higher internal resistance than a new battery. It is damaged if it has been fully charged for longer than a few months, if it has been discharged too low or if it has had too many charge-discharge cycles.

While all the other answers are great and tell what I would tell you too (the battery voltage actually get's lower when there is a load), I'd like to add something:

The reason which is presented that the voltage drops is the "internal resistance". I want to mention that the model of an internal resistance is JUST A MODEL that works very well in modelling the properties of a voltage source, while at the same time being simple and easy to calculate.

In reality, it's more complicated. The resistance of the internal components of the battery that current has to pass through (I intentionally don't call those "internal resistance", because this is a term from the above mentioned model) plays a role, but it's not the only role.
In most batteries there is a chemical reaction going on that separates charges at some border layer. This chemical reaction follows the laws of statistical physics. It comes to halt when the (chemical equilibrium is reached. The separation of charges generates the voltage that you can measure, and this voltage is a factor in the chemical equilibrium (the higher the voltage is, the less separation occurs to create a new pair of separated charges).
When you attach a load now, you take away charges at constant intervals (because there is an electrical current). If the system reaches the equilibrium situation now, the amount of separated charges and voltage will be less (because more charges have to be created).

Battery voltage doesn't usually drop just because there's a load connected. But the measured voltage does tend to drop

Here's what you've got to know about voltage measurements

A voltmeter uses a resistor with a very high resistance. Ideally, it's infinite. The voltmeter measures the voltage across this resistor.

So when you connect a battery to your voltmeter, the internal resistance of the battery is insignificant compared to the voltmeter's resistance. So most of the voltage drop occurs across the voltmeter's resistance and not the battery's internal resistor. Hence, you measure the right voltage.

However, your microcontroller may have a resistance that's not too high.
If the battery had an internal resistance of say, 1 milli ohm, and the voltmeter was using a 24000 ohm resistor, this error is expected.