Recently-ish, Critical Role has introduced a concept that I had never heard of before: passive Insight. Now, it makes sense to me conceptually, as well as being useful for that table and their trigger-happy Insight checks, but I had never heard of it.

Where did this concept originate? Is it from a previous edition, or is it one of Matthew Mercer's homebrew rules?

The Player's Handbook has a section for "Passive Checks" for all ability scores

These rules are found on page 175, entitled "Passive Checks":

Passive Checks

A passive check is a special kind of ability check that doesn't involve any die rolls. Such a check can represent the average result for a task done repeatedly, such as searching for secret doors over and over again, or can be used when the DM wants to secretly determine whether the characters succeed at something without rolling dice, such as noticing a hidden monster.

Here's how to determine a character's total for a passive check:

10 + all modifiers that normally apply to the check

If the character has advantage on the check, add 5. For disadvantage, subtract 5. The game refers to a passive check total as a score.

For example, if a 1st-level character has a Wisdom of 15 and proficiency in Perception, he or she has a passive Wisdom (Perception) score of 14.

The rules on hiding in the "Dexterity" section below rely on passive checks, as do the exploration rules in chapter 8.

In the player's handbook itself, the only explicit use of passive checks is on Perception checks, to handle whether characters who are not actively searching for something will notice a hidden object or creature. But in theory, this can be applied to any check.

So in this context, a passive Insight check allows a creature who is very bad at being deceptive (in general or in this specific circumstance) to immediately be called out by the DM. This can simplify play in more mundane settings, and while still allowing players to take actions if they have prior reasons to suspect the behavior of the character they are interacting with.

There are general rules for "passive checks" in 5e's skills chapter (Player's Handbook p.175), which "passive insight" is really just a specific implementation of. That didn't start with 5e, though. In 4th Edition, they had a general 'passive skills' concept ("Checks without rolls") that was along the lines of "you're always taking 10 with some skills". Insight was one of those. But the idea may predate even that, since there's long been an issue of knowing that you rolled a 2 on your perception or insight (or equivalent skills by other names) rolls and being aware that the information you got isn't trustworthy.

I wouldn't really call this Mercer's homebrew, both because it's kind of in the book, and because it's hardly Mercer's in specific, as (based on googling) there were discussions of Passive Insight in 5e going back as far as when the game was still being called "DnD NEXT".

You could say that passive skills are descended from the concept of "taking 10 or 20" in Third Edition. It's not entirely a clean match for either one, though.

The basic idea behind "taking 10" was that if you weren't in danger or being distracted, you could act more slowly and carefully. In situations like these, in lieu of rolling the die you could simply accept the same result as if the die had come up 10: it takes longer in game time, but it's more reliable than a die roll would be.

There was also a related concept of "taking 20", which you could use if there was also no time pressure and no penalty for failure. The character just tries over and over again until they get it right, and the result is the same as if the die had come up 20 (but this isn't considered a natural 20, in cases where that matters). This takes much longer in game time, and it can't be used in as many situations as taking 10, but it's as reliable as you can get.

Passive checks and skills don't quite line up with this: there is no equivalent to "taking 20", and the situations in which you can "take 10" are different. But the general idea of taking a reliable but not necessarily optimal result, in lieu of rolling the die, is the same.

Passive ability checks are explained in chapter 7 of the 5e Player's Handbook. While passive perception is the most commonly used passive check, any ability check can be made passive that the DM desires.